Introduction

Let's imagine that we're interested in testing Zipf's Law in some of our favorite books.

Zipf's Law

The most frequently-occurring word occurs approximately twice as often as the second most frequent word. This is Zipf's Law.

The texts we want to analyze are in .txt format:


ls inputs/txt/*.txt
 347.1k  10/20/21 21:14  abyss.txt         
 321.9k  10/20/21 21:14  isles.txt         
1001.3k  10/20/21 21:14  last.txt          
 343.5k  10/20/21 21:14  sierra.txt

With the user-written command wordfreq.ado and a helper ado-file countWords.ado, we have written a do-file count_isles.do that takes the file inputs/text/isles.txt as an input and returns outputs/data/dta/isles.dta, a file with the frequency of each word in the input file. First, we count the frequency of words in the file isles.txt:


. do code/count_isles.do

. version 16.1

. quietly include profile.do 
No resource locations have been stored with whereis

. 
. countWords, inputFile("isles.txt") outputFile("isles.dta")  
file outputs/data/dta/isles.dta saved

. 
. exit 

end of do-file

(If you look at countWords.ado, you will notice that, to keep our do-files and this document tidy and simple, we have hard-coded some paths to our inputs and outputs. This is not good programming practice.)

Let's look at the top 5 lines in the output file isles.dta:


. use outputs/data/dta/isles.dta, clear

. li in 1/5

     +------------------------+
     | word   freq      share |
     |------------------------|
  1. |  the   3315   .0625483 |
  2. |   of   2185   .0412272 |
  3. |  and   1530   .0288685 |
  4. |   to   1323   .0249627 |
  5. |    a   1132   .0213589 |
     +------------------------+

. clear

We can do the same for another book, this time in the file abyss.txt:


. do code/count_abyss.do

. version 16.1

. quietly include profile.do 
No resource locations have been stored with whereis

. 
. countWords, inputFile("abyss.txt") outputFile("abyss.dta")  
file outputs/data/dta/abyss.dta saved

. 
. exit 

end of do-file

. use outputs/data/dta/abyss.dta, clear

. li in 1/10

     +------------------------+
     | word   freq      share |
     |------------------------|
  1. |  the   3538   .0588284 |
  2. |  and   2524    .041968 |
  3. |   of   1715   .0285163 |
  4. |    a   1451   .0241266 |
  5. |   to   1375   .0228629 |
     |------------------------|
  6. |   in   1119   .0186063 |
  7. |    i    895   .0148817 |
  8. |   it    627   .0104255 |
  9. |  was    622   .0103424 |
 10. |  for    601   .0099932 |
     +------------------------+

. clear

Now we want to visualize the results. We have written a do-file plot_isles.do that reads in isles.dta file and, with a helper ado-file plotCounts.ado, plots the 10 most frequently-occuring words in a bar chart, which it saves in isles.png.


. qui do code/plot_isles.do

We can do the same for abyss:


. qui do code/plot_abyss.do

Now we can test Zipf's law for these books. We have written a do-file testZipf.do that uses the files of counts we have created to produce an output file with the ratio of the first to the second for each input file.


. qui do code/testZipf.do

. type outputs/tables/testZipf.txt
Book    First   Second  Ratio
isles   3315    2185    1.5171624
abyss   3538    2524    1.4017433

This is reasonably close to Zipf's law.

Our workflow is as follows:

  1. Read in a data file (text in this case).

  2. Analyze this data file.

  3. Save the results of the analysis to a new file.

  4. Plot a graph of the results of the analysis

  5. Save the graph as an image for later inclusion in a paper or other document

  6. Repeat 1-5 for each of our input datasets

  7. Combine results across datasets into a summary table

So far, we have been conducting steps 1-6 of this analysis one file at a time by running our do-files count_isles.do, count_abyss.do, plot_isles.do and plot_abyss.do on the command line, then conducting step 7 by running testZipf.do, again from the command line. This is fine for just two input files, but could become unwieldy if the number of input files or the number of steps in the workflow increased.

The standard solution in Stata is to write a "master" do-file to run this process. Here is an example, which we have named master.do:


version 16.1
// USAGE: do code/master.do
// to produce plots for isles and abyss
// and summary table for Zipf's law tests


// take the input file corresponding to each book, create frequency counts
quietly {
	do code/count_isles.do
	do code/count_abyss.do
}

// for each book, plot our graph in png format
quietly {
	do code/plot_isles.do 
	do code/plot_abyss.do
}

// combine results and summarize
quietly {
	do code/testZipf.do 
}

type outputs/tables/testZipf.txt

We can run the script again and see that the output is unchanged:


. qui do code/master.do

. type outputs/tables/testZipf.txt
Book    First   Second  Ratio
isles   3315    2185    1.5171624
abyss   3538    2524    1.4017433

Having a script (often called a master do-file) is a useful tool for reproducible results:

  1. We have a record of our workflow

  2. We can reproduce the full analysis with one command, master.do

However, it has a few important shortcomings, which we will illustrate using an example.

Suppose we wanted to narrow the bars in the graphs produced by plotCounts.ado. We edit the twoway command in plotCounts.ado, replacing barwidth(0.8) with, say, barwidth(0.6).

Now we want to recreate our figures. This is easily done by running master.do again.

This will work, but what if the analysis steps took a very long time? Our do-files (and the ado-files they use) to produce our word counts have not changed, and our input files have not changed, so we know that our datasets of word counts (isles.dta and abyss.dta) will not change if we re-run our analysis steps. We should not have to re-run our full analysis just to make some changes to our graphs.

One common approach is to manipulate the master.do script. For example, if we knew that our input files and our word-counting program had not changed, we could comment out the relevant lines in master.do. That is, we could replace


do code/count_isles.do
do code/count_abyss.do

with


// do code/count_isles.do
// do code/count_abyss.do

so that master.do will skip over these lines and not re-run the analysis.

For a simple workflow, this is probably OK, but it can be a source of errors in more complicated workflows or in collaborative work. For example, in a complicated workflow, you might update an input file, but then forget that you had commented out the analysis lines in your script. Or, in a collaborative project, a co-author may have made a change to countWords.ado that you have not noticed.

In order to avoid both types of error -- re-running analyses that are not necessary or failing to update analyses when needed -- what we really want is an executable description of our pipeline that allows software to do the tricky part for us: figuring out what steps need to be rerun.