A summary, of what we believe is the most comprehensive analysis of the subject matter to have been undertaken is provided below:

Benjamin Graham developed an investment strategy that involved purchasing securities for less than their *“current-asset value”*, *“a rough index of the liquidating value”. *We uncovered ten research papers that examined the returns achieved by investing in such securities which were conducted over a number of decades and across various geographies. In general, the research found that the strategy generated a remarkable level of outperformance. Our objective was to answer the question, “Could the returns reported in the research have been achieved by an investor in practice?”. To meet this objective, we developed a methodology to analyse the evidence and determine its reliability. Subsequently, we found that each of the studies suffered, or may have potentially suffered, from a number of biases which adversely impacted the reliability of the results contained therein. We therefore concluded that, in essence, a practitioner could **not** have achieved the returns reported in the research. Lastly, we briefly discussed the implications of our findings for practitioners of the investment strategy, as well as for evidence based investors at large.

The focus of this analysis is the research paper “Ben Graham's Net Nets: Seventy-Five Years Old and Outperforming” by Tobias Carlisle, Sunil Mohanty, and Jeffrey Oxman[i] published in 2010. The objective of the paper was to provide an update on the research conducted by Henry R. Oppenheimer in his 1986 study titled, “Ben Graham’s Net Current Asset Values: A Performance Update.” (We examined Oppenheimer’s research in our article, “An Analysis of Benjamin Graham’s Net Current Asset Values: A Performance Update”.) Following the methodology adopted in Oppenheimer’s study, the authors selected securities that were trading at no more than two-thirds of their Net Current Asset Value (NCAV). Their 25 year examination period ran from 31 December 1983 to 31 December 2008 and they focussed on US listed securities.

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the NCAV method of investing.

We have previously published a number of posts with regard to securities trading at less than their Net Current Asset Value including:

1. ,An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”

2. ,An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”

3. ,An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London”

4. Analyzing Deep Value in the Eurozone

5. What Has Worked in Investing (Tweedy, Browne) – Examining Net Nets

6. Examining Saudi Arabian Net Nets

7. Examining Greenblatts “How the small investor can beat the market”

8. Examining “An Empirical Analysis Of Ben Graham's Net Current Asset Value Rule”

9. An Analysis of “Testing Benjamin Graham’s net current asset value model”

- Valuation metric:

*“…sum of all liabilities and preferred stock and subtracted it from current assets; this result was then divided by the number of common shares outstanding.”*

i.e. Net Current Asset Value per share = (Current Assets – (Total Liabilities + Preferred Shares))/Common Shares Outstanding

*“…bought a security if its November closing price was no more than two-thirds of its NCAV”*

- Weighting of holdings:
*“equal-weighted”*

- Purchase/rebalance date:
*“December 31st”*

- Holding period:
*“held for one year”*and*“held for 30 months”*

Studies can suffer from a number of issues which reduce their reliability. Below we address potential concerns around sample size, return calculation methodology, data sources and common biases that may afflict research:

1. Sample size, exchanges and firm characteristics

From 1984 to 2008 the number of firms meeting the necessary criteria ranges from 13 to 152 with an average of almost 55 firms. Relative to other studies, this study contains a far larger sample of firms.

Exhibit 1 is reproduced below:

No industries or sectors were excluded from the study. No minimum market capitalization was specified. Consequently, the inclusion of the smallest firms could have unduly influenced results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable.

2. Return calculation methodology

The authors calculated and reported the “Mean Returns”. In academic research (such as this) returns are often measured by calculating the arithmetic mean return (as opposed to the geometric mean return) - this appears to be the case in this study. It should be noted that in a dependant return series that exhibits volatility (like stock returns) the arithmetic mean will, as a matter of mathematical law, overstate returns relative to the more practitioner oriented geometric mean. The magnitude of the potential divergence between the two measures is unknown given the data made available in the study.

We will examine this further in the next section.

3. Data source

Under the “Data and Methodology” section no mention of a data source was made. In the “Results” section the following was stated: *“While Oppenheimer uses the Ibbotson Small-Firm Index, we use the smallest decile of stocks traded in the CRSP database.” *This implies the study was conducted using Center for Research in Security Prices (CRSP) data. Presumably then, the fundamental data came from Compstat based on the “CRSP/Compustat Merged Database”, however, this is *assumed* and not stated in the study.

The Center for Research in Security Prices (CRSP) database is considered to be the gold standard for research.

4. Survivorship bias

The authors presumably used the CRSP database which includes data on delisted stocks. However, the delisted company data needs to be incorporated into the return data. An algorithm that mergers the CRSP delisting information into the returns is identified in the paper “Delisting Returns and Their Effect on Accounting-Based Market Anomalies,” by Richard Price, William Beaver and Maureen McNichols[ii]. However, it is not clear if the delisting data was merged into the return data used in this study.

Given that no specific mention was made with regard to controlling for survivorship bias it is possible, though not probable, that the study may have suffered from such a bias.

5. Look Ahead bias

They *“bought a security if its November closing price was no more than two-thirds of its NCAV”.* Presumably, the November NCAV was based on financial statement data related to the previous 30 September (or perhaps previous 30 June). *“We assume all stocks are purchased on December 31st”. *

For clarity, a firm needed to meet the 2/3 of NCAV criteria based on its price on 30 November, however, the returns thereto were measured using its market price at 31 December.

Based on the above the study does not appear to suffer from look-ahead bias.

6. Time period bias

The study spans 25 years and we classify this as a “more reliable” period.

For reference:

- < 10 years; inadequate/unreliable
- 11 to 20 years; somewhat reliable
- > 20 years; more reliable
- > 40 years; most reliable

7. Human error

There is nothing specified in the research methodology that would make us believe this study is at greater risk of suffering from human error.

8. Journal rating/credibility[iii]

This study was not, to our knowledge, published in a top tier academic journal and therefore cannot be granted the “additional credibility” that may come with such publication.

**Reliability Assessment**: Despite the relatively long time period (25 years) and relatively large sample size, due to the presentation of the arithmetic mean return (i.e. “Mean Return”) we would not place reliance on the results of the study from a practitioner’s standpoint. Also of concern is that no minimum market capitalization was specified for the securities examined. Consequently, the inclusion of the smallest firms may have unduly influenced results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable.

While this study purported to be an update of Oppenheimer’s 1986 study unfortunately it did not contain the corresponding tables. Specifically, the most useful table from Oppenheimer’s study was Table IV which displayed the return in each year of the study and presented the “Annual Geometric Mean Return”. The “Annual Geometric Mean Return” represented the actual return potentially achievable in practice (gross of commissions and taxes).

*“Exhibit 2 summarizes the results for the 25-year period of the study”*, which deals with the 12 month holding period:

We note that the table heading states it is a “26” year period, however the study covered 25 years.

They present “monthly” “Mean Returns” (i.e. arithmetic mean). Exhibit II does indeed replicate the data presented in Table II of Oppenheimer’s study. Furthermore, based on reconciliation work carried out in “An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”, it reinforces our view that the arithmetic mean returns have also been presented in this case.

*“The mean monthly return on stocks meeting the NCAV rule in the period we examined, December 31, 1983 to December 31, 2008, was 2.55%. The mean monthly returns for the NYSE-AMEX and Small-Firm indices were 0.85% and 1.24% respectively. This indicates an outperformance by the NCAV portfolio over the NYSE-AMEX Index of 1.70% per month, or 22.42% per annum and an outperformance over the Small-Firm Index of 1.31% per month, or 16.90% per annum.”*

*“Panel C presents results for nine consecutive sub-periods, eight of which are of approximately equal length, and the final of which is incomplete.” *It is unclear how the aforementioned reconciles to the data in Panel C which appears to commence and end with a 2 year period.

30 month holding periods are examined and presented in Exhibit III (reproduced below). The authors state *“Graham suggested that a 30-month holding period was appropriate for the NCAV investment practice, rather than the twelve month period we have assumed thus far. Since the 30-month portfolios overlap each other, we create each using unique stocks. Thus, no two portfolios contain the same company’s shares. We assume, as above, that the portfolios are created on December 31st of each year, and then held for 30 months. However, portfolios created in 2006 have only 24 months of observations, and portfolios created in 2007 have only 12 months of observations. Rather than eliminate these years from the study, we report results with the above caveat in mind.”*

At first glance Exhibit III, through its measurement of the “Terminal Wealth of $10,000” appears to be the representation of the *actual dollar return* achievable by an investor in practice (i.e. geometric mean/compounded return). However, like we discovered in An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London” and What Has Worked in Investing (Tweedy, Browne) – Examining Net Nets researchers “compounded” the arithmetic mean in attempt to put the returns in more “meaningful terms”. Indeed, we believe this to be the case in this study also. For instance, the 1983 monthly mean return of 3.59% when compounded for the stipulated 30 months with a starting value of $10,000 yields (rounding aside) the reported ~$28,846.24 (10,000*(1.0359)^30). When 3.59% is compounded monthly it results in an *annualized* (i.e. compounded) return of 55.7% ((28,846.24/10,000)^(12/30)-1). Similarly, in 1993 the reported monthly mean return was “7.97%”. When $10,000 is compounded for 30 months at 7.97% it yields (rounding aside) the reported ~$99,867.78 (10,000*(1.0797)^30). When 7.97% is compounded monthly it results in an *annualized* (i.e. compounded) return of 151.1% ((99,867.78/10,000)^(12/30)-1).

The implied returns are as extraordinary as they are illusionary!

*“According to Oppenheimer, Graham frequently recommended that it was best to select NCAV securities that had positive earnings and paid a dividend. Oppenheimer’s findings seem to contradict this advice. He found that firms operating at a loss seemed to have slightly higher returns and risk than firms with positive earnings. Firms with positive earnings paying dividends provided a lower mean return than portfolios of firms with positive earnings not paying a dividend, but had a lower systematic risk. These findings led Oppenheimer to conclude that choosing only firms that have earnings and pay a dividend will not help the investor.*

*Our results, presented in Exhibit 4, support Oppenheimer’s conclusion. Firms with positive earnings generated monthly returns of 1.96%. By contrast, firms with negative earnings generated monthly returns of 3.38%. Firms with positive earnings paying dividends in the preceding year provided monthly returns of 1.48%, a lower mean return than portfolios of firms with positive earnings with no dividend paid in the preceding year (2.42%), but did have a lower systematic risk.”*

In our analysis of Oppenheimer’s paper, (“An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”) we identified that the research had been conducted with data that *excluded* dividends. Therefore, the conclusion reached in that study, *“Choosing only firms that have…. a dividend does not help the investor”* was not considered reliable. Furthermore, as data excluding dividends was also used to measure the returns of “positive” and “negative” earning firms, no reliable conclusion in relation to the impact of earnings on the return of the securities examined could be ascertained from the results of Oppenheimer’s study either. Given the flaws in Oppenheimer’s study we do not consider the results reported in this study as reinforcing his conclusions. Rather, the results in this study need to be viewed in isolation of Oppenheimer’s findings.

Exhibit 4 is reproduced below:

Having established that Oppenheimer’s results were compromised due to the exclusion of dividends we proceed by examining the same issue in this study.

The authors state, *“Monthly returns are presented for the NCAV portfolios against various benchmarks, and sorted by earnings record and dividend payments. Rpt and Rmt are the NCAV portfolio and benchmark returns respectively….For all benchmarks, we use returns including dividends, except for the following: S&P 500, AMEX, and Nasdaq. These items are returns without dividends.” *Why total returns (i.e. including dividends) weren’t used with regard to the S&P 500, AMEX, and Nasdaq is unknown. While portfolios may have been sorted by earnings and dividend payments it remains unclear whether the data used to measure the returns incorporated the dividend payments themselves.

*“The results in this section indicate a rational connection between risk and return. Dividend-paying firms are viewed as less risky because the dividend signals to shareholders that managers believe the future cash flows of the firm are stable enough to accommodate an ongoing dividend.”*

To accept the above conclusion an investor must also accept that “risk” is represented by the various “Risk Adjusted” measures presented, along with their implicit association with price volatility. Indeed, the acceptance of “beta” as an appropriate measure of risk adjusted returns runs contrary to the “low-beta anomaly”[iv].

Also, of concern is that no hypothesis (i.e. preceding the testing) was provided (in either study) for such results. Absent a more sound economic rationale for why positive earning non-dividend paying firms, 2.42% (and negative earning firms, 3.38%) would generate higher returns than positive earning dividend paying firms, 1.48% (and positive earning firms, 1.96%), data mining or the exclusion of dividends from the return data may provide greater explanatory power for the reported results compared to that provided within the study.

Exhibit 5 and the corresponding narrative is reproduced below:

*“Another question an investor may have is: does the depth of discount affect future returns? We have shown so far that the NCAV rule is an extreme form of value investing. A logical question comes up: do the deepest discounted NCAV stocks provide the highest returns in the future? To examine this, Oppenheimer calculated for each security its purchase price as a percentage of NCAV, and divided the population into quintiles according to this variable.”*

*“Adopting the same method, we analysed mean returns and risk-adjusted performance. The results are presented in Exhibit 5. Quintile 1 contains the fifth of the firms that have the highest discount, and Quintile 5 contains the firms trading closest to two-thirds of NCAV. With one caveat, our findings generally support Oppenheimer’s conclusion: the returns are higher for firms with higher discounts to NCAV. The caveat is as significant as it is perplexing: securities in Quintile 1– those with the lowest purchase price to NCAV – have the lowest returns. As noted earlier, we have eliminated as outliers firms with stock prices less than one percent of the NCAV per share, so we do not believe outliers are driving this result.”*

*“In results not reported here, we plot returns for each rank by year. No pattern exists in the ranked returns. Although the returns in rank 2 and rank 3 tend to be the highest, this is not always the case. So, we can say that on average there is a mild positive relationship between the depth of discount and future returns, there is such variability year-over-year that we cannot suggest this is a reliable rule.”*

Some food for thought pertaining to the reported results: in general, value investing research[v] shows returns that are consistent with the relative “cheapness” of the underlying portfolios examined. Firms trading at less than 2/3 of NCAV would, almost certainly, consist of firms that in aggregate sit in the bottom decile of price to book (or by academic convention, top decile of book to market). Perhaps then, sorting these firms again based on discount to NCAV possesses limited utility; rather, other metrics (e.g. “quality measures”) may be required to eliminate the relatively poor performers. Indeed, in “Analyzing Deep Value in the Eurozone” we also observed that the firms that traded in the cheapest decile sorted by *relative* price to NCAV produced *lower* returns that those in the second cheapest decile.

As such then, we have no definitive explanation regarding the above results and only proffer thoughts for further contemplation.

The authors attempt to explain the “excess returns” by regressing a number of factors against the NCAV returns. For interested readers we recommend referring to the underlying paper, nonetheless, we cover the key statements below:

**Market Risk**

Unsurprisingly, they find *“more than mere market risk explains the returns on the NCAV portfolios”*.

**Fama-French three factor model (Market/Size/Value)**

*“Applying the Fama-French three factor model to our data confirms the importance of the small-firm effect. All three factors are statistically relevant as explanatory variables, but the small firm effect is the largest factor. *

They proceed and subsequently state: *“However, there is still an economically significant excess return (alpha) of 1.67% per month. Assuming monthly compounding, this yields an excess return of 21.99% per year after controlling for market risk, the small-firm effect, and the book-to-market effect.” *Why they “assume” compounding, when the returns calculated are actually “average” (i.e. arithmetic mean) returns remains unknown. However, the statement implies that the returns presented are “compounded returns” (i.e. geometric mean returns) which is misleading.

*“[T]he value premium adds a minor* [our emphasis]

It is interesting to note that purchasing firms trading below a NCAV (essentially an extreme form of price to book value) does *not* provide significant explanatory power, at least according to the regression analysis conducted.

**Long Term Price Mean Reversion**

*“We apply the long-term reversal factor to our data …. While positive, the factor is not economically or statistically significant. Therefore, long-term reversal does not appear to explain the excess returns on NCAV portfolios.”*

**Momentum**

*“Since NCAV stocks are recent losers, we should expect the momentum factor to be negatively related to the returns on NCAV portfolios, and it is. It is also statistically significant – see Exhibit 7, Model 1 in Panel A [not reproduced here].” “What is especially interesting is that inclusion of the momentum factor washes out the significance of the value factor (HML), and causes it to change sign. This indicates that the momentum factor is an important variable in explaining returns on NCAV portfolios.”*

**Liquidity**

*“We add the liquidity measure in with the Fama-French 3 factors and the momentum factor, which was previously found to be significant, and estimate Model 5. Now we find HML and MOM are insignificant, and so discard these factors and estimate Model 6. The fit of this model is the highest of all those estimated, and so we are confident that the market risk, small-firm effect (SMB) and liquidity factors (ILLIQ) are the only* [our emphasis]

**The January Effect**

*“The January effect in our portfolios accounts for 10% of returns in the year, while the other eleven months account for about 2% each. Thus, the January effect is nearly half of the whole year’s return. This does not mean, however, that the January effect drives returns on the NCAV portfolios.”*

Indeed, the results of the regression analysis may elicit a range of reactions from readers. Some readers may consider the results to be “objective and instructive” while a review of the results by other readers may evoke the quote “Lies, damned lies, and statistics”!

We leave it to readers to ascertain the utility, or otherwise, of that presented by the authors in this section.[vi]

“Ben Graham's Net Nets: Seventy-Five Years Old and Outperforming” was a study that covered the 25 year period from 31 December 1983 to 31 December 2008 and it focussed on US listed securities trading below two-thirds of their NCAV. In addition to examining the returns over a 12 and 30 month holding period respectively, the authors also attempted to explain the drivers of the observed excess returns by regressing various factors against the returns generated by the NCAV portfolios.

Their conclusion states *“The results are as clear as they are compelling: Seventy five years on, Graham’s NCAV rule continues to identify securities that generate above-market returns.”*

When looking at the findings reported in the paper from a practitioner’s point of view, our conclusion is different. The “mean returns” reported in the study were calculated as the arithmetic mean of returns which would have overstated the actual returns potentially achievable by an investor in practice. Without specification of the geometric mean return, which reflects practitioner reality, we cannot be certain of the actual return achieved. In addition, no minimum market capitalization was specified for the securities examined and consequently their inclusion may have unduly influenced the results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable.

For the practitioner not afraid to be confronted with a conclusion that may be inconsistent with their prior beliefs, we offer our version of the authors conclusion based on our extensive analysis of the study:

*“The results are as [unclear] as they are [overstated]: Seventy five years on, Graham’s NCAV rule continues to identify securities that [may] generate above-market returns, [we simply do not know based on the methodology adopted in this study].”*

**Notes:**

[i] We contacted the corresponding author on a number of occasions to seek clarity on aspects of the study and never received a response, nor did we receive a response from the second listed author and the first listed author did not have in his possession the underlying data which we wished to review. [ii] Quantitative Value - Carlisle, Tobias, Gray, Wesley Ph.D., Chapter 10.

[iii] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

[iv] For research on Low-Volatility/Low-Beta see here: https://alphaarchitect.com/category/architect-academic-insights/factor-investing/low-volatility-investing/ [v] For further value investing research see here: https://alphaarchitect.com/category/architect-academic-insights/factor-investing/value-investing/

[vi] The same research team (Tobias Carlisle, Sunil Mohanty, and Jeffrey Oxman) authored the paper “Dissecting the Returns on Deep Value Investing” (2012) (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1928694 ) in which they examined numerous factors in an attempt to “explain the returns” to firms trading below NCAV (albeit with a variation of the valuation methodology applied in this study).

]]>

The focus of this article is the research paper “Testing Benjamin Graham’s net current asset value model” by Chongsoo An, John J. Cheh , and Il-woon Kim published in Journal of Economic & Financial Studies (2015). The objective of the paper was to examine the performance of securities that were trading at less than their Net Current Asset Value (NCAV) during the 14 year period from January 2, 1999 to August 31, 2012 in the US market.

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the NCAV method of investing.

We have previously published a number of posts with regard to securities trading less than their Net Current Asset Value including:

1. ,An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”

2. ,An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”

3. ,An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London”

4. ,Analyzing Deep Value in the Eurozone

5. What Has Worked in Investing (Tweedy, Browne) – Examining Net Nets

6. ,Examining Saudi Arabian Net Nets

7. Examining Greenblatts “How the small investor can beat the market”

8. Examining An Empirical Analysis Of Ben Graham's Net Current Asset Value Rule

- Valuation metric:

*“According to Benjamin Graham, net current assets are defined as current assets minus total liabilities (and preferred stock if any)”*

i.e. Net Current Asset Value = Current Assets – (Liabilities + Preferred Shares)

While the authors did not expressly state that they used the formula specified above we assume they did so.

*“To test the Graham’s net current asset value method, the ratio of the net current asset value to market value (NCAV/MV) was employed in this study as the criterion in selecting stocks…“all securities that satisfied the primary condition “NCAV/MV > market price” were selected.”*

- Weighting of holdings: Not specified.

- Purchase/rebalance date: Not specified.

- Holding period:
*“the returns of different holding periods were tested: one year, six months and four months”*.

Studies can suffer from a number of issues which reduce their reliability. Below we address potential concerns around sample size, return calculation methodology, data sources and common biases that may afflict research:

1. Valuation criteria, sample size and firm characteristics

*“The analysis of stock performance was conducted in two stages. First, all securities that satisfied the primary condition “NCAV/MV > market price” were selected. The 126 selected securities were then pooled into three different portfolios using the secondary condition “market price×N; N=1, 2, & 5” as follow:*

*Portfolio 1: NCAV/MV > market price×1 *[i.e. trading at less than NCAV]

*Portfolio 2: NCAV/MV > market price×2 *[i.e. trading at less than 50% of NCAV]

*Portfolio 3: NCAV/MV > market price×5 *[i.e. training at less than 20% of NCAV]

*The final sample size for each portfolio was 84 firms, 32 firms, and 10 firms for Portfolio 1, Portfolio 2, and Portfolio 3, respectively. N was the weighing factor for the market price. For example, N=5 indicated that the NCAV/MV ratio was higher than five times the market price.”*

There is a discrepancy between the portfolio formation criteria and the narrative; firms that met the criteria for inclusion in Portfolio 3 would also meet the specified criteria for Portfolio 2 and Portfolio 1. However, given the number of firms that were specified to be contained in each portfolio the authors actually tested firms trading at *greater than* the specified NCAV/MV cut off for the relevant portfolio, but *less than* the NCAV/MV cut off for the next portfolio.

With regard to sample size, only 126 firms traded at less than their NCAV (i.e. NCAV>MV) in the 14 year test period. Furthermore, the sample size was not specified by year. Presumably, there would have been years where few firms met the criteria as markets rose, and as markets fell more firms would have met the necessary criteria. For perspective, the approximate “average” number of holdings for Portfolio 1, 2 and 3 over the 14 years was 6, 2 and 1 (rounding up) respectively. Consequently, one needs to be cautious when interpreting results, especially those pertaining to Portfolio 2 and 3.

Significantly, no minimum market capitalization was specified. Consequently, the inclusion of the smallest firms can distort results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable.

2. Return calculation methodology

The authors calculated the “Annualized returns” (i.e. geometric mean) and the “Average returns” (i.e. arithmetic mean).

It should be noted that in a dependant return series that exhibits volatility (like stock returns) the arithmetic mean will, as a matter of mathematical law, overstate returns relative to the more practitioner oriented geometric mean.

We will examine this further in the next section.

3. Survivorship bias

The authors used Portfolio123 who state that their financial data is free from survivorship bias.[i]

4. Look Ahead bias

As in the case of survivorship bias Portfolio123 state that their data is free from look ahead bias.[ii]

5. Time period bias

The study spans 14 years and we classify this as a “somewhat reliable” period.

For reference:

· < 10 years; inadequate/unreliable

· 11 to 20 years; somewhat reliable

· > 20 years; more reliable

· > 40 years; most reliable

The authors also acknowledge concerns with regard to the time period studied stating *“extending the study period and adding more sample firms will definitely improve the relevance of the study.”*

6. Data source

“*As the initial sample, we used all stocks in Portfolio123 (about 6,000 firms) which are supplied by Compustat, Standard & Poors, S&P Capital IQ, and Interactive Data.*

Presumably the data sources covered stocks listed across all the major US exchanges.

No mention was made with regard to the treatment of dividends; presumably returns included dividends in light of the reputable data sources used in the study.

7. Human error

There is nothing specified in the research methodology that would make us believe this study is at greater risk of suffering from human error.

8. Journal rating/credibility[iii]

This study was not, to our knowledge, published in a top tier academic journal and therefore cannot be granted the “additional credibility” that may come with such publication.

**Reliability Assessment**: Due to the relatively short time period (14 years), sample size concerns and the absence of a minimum market capitalization requirement for the securities examined the study does not appear to be reliable.

The authors calculate and present the “annualized returns” (i.e. compound annual growth rate/geometric mean) which represent the gross returns a practitioner could have actually achieved.

*“The annualized returns of three portfolios and S&P 500 during the study period are presented in Exhibit 2. Annualized returns are the returns that should have been realized every year to earn total returns during the study period. Theoretically, the stocks with a higher NCAV/MV value should be generating annualized returns higher than the stocks with a low value. The results of this study, however, are mixed. Portfolio 1 (4.15%) and Portfolio 2 (2.49%) beat the market with a big margin as shown in Exhibit 2, while Portfolio 3 (0.51%) does not do well compared to the S&P500 (0.96). It is also puzzling to see in Exhibit 2 that the returns are decreasing as the value of N is increasing from 1 to 2 and to 5. We believe that these mixed results are due to the fact that the number of firms in each portfolio is decreasing from 84, to 31 and to 10. As the sample size is getting smaller, the results of the study are getting less reliable and sometimes inconsistent.”*

It is interesting to note the that the Portfolio returns were in the opposite sequence to that which was expected. *“It was expected that the stocks with a higher NCAV/MV value (e.g., N=5) would be generating returns higher than the stocks with a low value (e.g., N=1).”*

One wonders how the returns impacted the narrative; had the returns been in accordance with expectation would less emphasis have been placed on the small sample size and instead would the returns have been taken as “evidence” that “cheapness” drives returns?

Sample size concerns aside, it is noteworthy that the very cheapest stocks did not even beat the market. We suggest that the results provide food for thought; perhaps cheapness alone cannot be relied on when dealing in securities trading below NCAV. Rather, further criteria may need to be considered when making investment decisions.

What is particularly interesting about these results is the low level of absolute return outperformance relative to the market when compared to other studies examining the returns to firms trading below NCAV. For example, Portfolio 1, the best performing portfolio, the absolute outperformance relative to the S&P 500 was just 3.19% (4.15%-0.96%). Furthermore, net of fees and commissions the absolute return outperformance would have been even lower. Indeed, the annualized return of three-month Treasury bills over the 14 year period was approximately 2.3%[iv]; a figure reasonably comparable to the net of fee and commission return likely to have been achieved. Furthermore, net of taxes (and effort) investing in firms with a NCAV > MV (i.e. trading at a discount to NCAV) appears to have been a forlorn endeavour in the US market over the test period.

The authors (with a focus on demonstrating the results of their “hedging strategy”) also present the “average” (i.e. arithmetic mean) returns. As mentioned in the “Reliability” section above, in a dependant return series that exhibits volatility (like stock returns) the arithmetic mean will, as a matter of mathematical law, overstate returns relative to the more practitioner oriented geometric. We examined this phenomena in more detail in An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”. Furthermore, we also discussed this point in An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London” along with how “confirmation bias” may impel an investor to believe the arithmetic mean could be used to “reasonably estimate” the geometric mean (i.e. the return potentially attainable in practice) and warned against such an endeavour.

This study demonstrates objectively the reason for such warning.

The “average” (i.e. arithmetic mean) returns are reported as follows:

Recall the annualized return (compound annual growth rate/geometric mean) for Portfolio 1, 2 3 and the S&P 500 was 4.15%, 2.49%, 0.51% and 0.96% respectively. In contrast the *“simple averages *[i.e. arithmetic mean]* of all rebalancing returns realized in backtesting using Portfolio 123 with one year holding period” *for Portfolio 1, 2 3 and the S&P 500 was 17.17%, 17.78%, 14.87% and 2.91% respectively.

Using Portfolio 1 as an example, that represents a 13.02% return differential. To illustrate the magnitude of that differential, compounding $100,000 at the Portfolio 1 annualized return rate of 4.15% yields $176,696 (100,000*(1.0415)^14-1) over 14 years (the length of the study period). In contrast, wrongly assuming that one could “compound” at the (arithmetic) average rate of 17.17% per Portfolio 1 would result in a mistaken belief that a terminal value of $784,537 (100,000*(1.1717)^14-1) was achievable. That represents a $607,841 (or 77.5%) difference in potential expectation.

*“As an attempt to improve the performance in the down market, a hedging strategy was implemented for each portfolio. The strategy was that the market condition should be favorable before any stocks were purchased. Understanding that the Federal Reserve Board tends to increase interest rates during a growth period and that the yield of corporate stocks would fall, the specific rule in this study is to buy stocks when the10-year Treasury yield are no higher than the yield 20 trading days ago.” (“CEY = TBY, where CEY is Current Estimated Yield (S&P current year estimates divided by S&P price) and TBY - Treasury Note Yield (10 year).”)*

This hedging strategy materially improved the annualized returns and reduced the standard deviation.

Prima facie the hedging strategy appears to possess utility. However, from our analysis of other studies such as An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”, An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London” and Examining Greenblatt’s “How the small investor can beat the market” we know the number of firms trading below NCAV tends to wax and wane. This tendency leads to periods of high and low concentration in the number of firms trading below NCAV. Given we do not know how many firms met the criteria for investment in each year of the study, it is possible (probable) that the majority of candidates were found in the minority of years. Consequently the proposed “hedging strategy” may have benefitted from providing a “buy signal” at an opportune time and without suffering from being return reducing in periods where a “false” buy signal was created due to the relatively low number of firms meeting the necessary criteria for investment. Indeed, with the overall small sample size the hedging strategy signal may not have had many opportunities to have its hypothesized efficacy tested robustly enough to be considered reliable. Furthermore, technically speaking, we would not consider the proposed “hedging strategy” as truly providing a “hedge”; rather it appears to be merely “market entry signal” without a corresponding “market exit signal” which would be required for it to be considered a “hedging strategy”.

The authors also suggest that further work is required on the hedging strategy stating, *“Using different benchmarks, new buy-and-sell rules can be created and tested on the relationship between earnings and market conditions.”*

The authors also examined the annualized returns achieved by Portfolio 1, 2 and 3 over three holding periods: 1 year, 6 months and 4 weeks. Observing the annualized returns “without hedging” (not reproduced here) we note that a consistent pattern does not emerge with regard to the various holding periods for any Portfolio. For instance, Portfolio 1 generated fluctuating annualized returns of 4.15%, 0.96% and 4.00% over the 1 year, 6 month and 4 week holding periods respectively.

“Testing Benjamin Graham’s net current asset value model” was an interesting 14 year study commencing in 1999 and concluding in 2012 that examined portfolios of US listed firms whose NCAV was greater than its “market value”(MV) (i.e. traded at a discount to NCAV). Despite the quantification of annualized returns, the study cannot be considered reliable as no minimum market capitalization was specified for the securities examined. Consequently, the inclusion of *all* firms may have unduly influenced the results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable. In addition, the particularly small sample size of securities meeting the necessary criteria was also of concern.

Interestingly, the authors also presented the arithmetic average returns which provided an objective lesson into why attempting to estimate the geometric mean (i.e. annualized return/compound annual growth rate) from the arithmetic mean is an inadvisable action. While Portfolio 1 generated an annualized return of 4.15% the practically unattainable arithmetic average return for the Portfolio was far greater at 17.17%.

With regard to return performance, the “best” performing portfolio, Portfolio 1, generated an annualized return (i.e. compound annual growth rate) of only 4.15%, an absolute excess return of just 3.19% over the S&P 500 which returned only 0.96% over the 14 year period. We say “only” and “just” due to both the relative and absolute outperformance reported in other studies (of variable reliability) examining firms trading below NCAV, as well as the historical return achieved by firms forming the S&P 500 index.

Reliability concerns notwithstanding, the study demonstrated that even over a 14 year period investing in firms trading below NCAV may not provide immunity from a low return environment - a sobering realization.

**Notes:**** **

[iii] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

[iv] http://pages.stern.nyu.edu/~adamodar/New_Home_Page/datafile/histretSP.html

]]>

The focus of this post is the research paper “An Empirical Analysis Of Ben Graham's Net Current Asset Value Rule” by Joseph D Vu published in The Financial Review (1988). The objective of the paper was to examine the performance of securities that were trading at less than Net Current Asset Value (NCAV) during the 8 year period from 1977 to 1984 on the New York Stock Exchange (NYSE) and American Stock Exchange (AMEX).

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the NCAV method of investing.

We have previously published a number of posts with regard to securities trading less than their Net Current Asset Value including:

- An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”
- An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”
- An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London”
- Analyzing Deep Value in the Eurozone
- What Has Worked in Investing – Examining Net Nets
- Examining Saudi Arabian Net Nets
- Examining Greenblatts “How the small investor can beat the market”

- Valuation metric:

*“The NCAV is defined as current asset minus all liabilities including long-term debt and preferred stock.” *

i.e. Net Current Asset Value = Current Assets – (Liabilities + Preferred Shares)

Stocks qualify for investment when they sell at (i.e. have a market capitalization) less than their NCAV. Unlike Graham, the author did not require any specific “margin of safety” relative to NCAV, e.g. 2/3 of NCAV.

- Weighting of holdings: Equal weight

- Portfolio formation:
*“stocks meeting the NCAV can enter the portfolio continuously at the end of each month, not on an arbitrary date such as December 31”. “This stock selection method … includes all stocks selling below NCAV at any time”.*

- Holding period: 2 years;
*“Each stock can enter the portfolio only once during a two-year period… For stocks that are acquired or liquidated, the final stock price is used in calculating monthly returns.”*

Studies can suffer from a number of issues which reduce their reliability. Below we address potential concerns around sample size, return calculation methodology, data sources and common biases that may afflict research:

1. Sample size and firm characteristics

As is often the case, the number of firms selling below NCAV relative to the number of firms in a market is relatively low. In the examination period (1977-1984) 107 firms sold for less than NCAV on the NYSE and AMEX. Noteworthy is that fact that of those 107 firms, 50 were found in 1977 alone, 0 in 1983 and just 2 in 1984. By deduction, that means from 1978 to 1982, 55 firms sold for less than NCAV (an average of 11 per year).

The mean market capitalization of firms examined was “$51.3 million”[i]. However, no minimum market capitalization was specified. Consequently, the inclusion of the smallest firms may have biased the results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable. The larger market capitalization relative to firms identified in other studies is likely a result of not including firms that traded over the counter as part of the assessable investment universe.

2. Return calculation methodology

“Raw Returns” are used to calculate investment performance. Raw returns are returns that are not adjusted for “risk” e.g. the Sharpe ratio is an example of a risk-adjusted return metric. In academic research (such as this) returns are often measured by calculating the arithmetic mean return (as opposed to the geometric mean return) - this appears to be the case in this study. It should be noted that in a dependant return series that exhibits volatility (like stock returns) the arithmetic mean will, as a matter of mathematical law, overstate returns relative to the more practitioner oriented geometric mean. The magnitude of the potential divergence between the two measures is unknown given the data made available in the study.

3. Survivorship bias

No mention was made with regard to controlling for “survivorship bias”.

However, given the authors used the physical “Value Line Investment Survey” from April 1977 to December 1984 (as opposed to a computerized data base that might not include delisted stocks) the data source would logically be free from survivorship bias.

4. Look Ahead bias

Controlling for look ahead bias was not specifically mentioned.

Prices were attained from the *“Center for Research in Security Prices (CRSP) monthly return tapes at the University of Chicago”*. While this is considered the “gold standard” for price data it doesn’t alleviate the possibility that prices retrieved from the database matched e.g. the date of underlying financial statements and therefore didn’t account for the delay in their public availability.

In the absence of an express statement addressing look ahead bias it remains a possibility.

5. Time period bias

The study spans 8 years and we classify this as an “inadequate/unreliable” period. However, as the authors mention that two stocks met the valuation criterion in 1984, and assuming the returns to those stocks were measured over the stipulated 2 year holding period, the study period may have been almost 10 years in length (i.e. April 1977 to December 1986).

For reference:

· < 10 years; inadequate/unreliable

· 11 to 20 years; somewhat reliable

· > 20 years; more reliable

· > 40 years; most reliable

6. Data source and treatment

As mentioned previously the “Value Line Investment Survey” was used for fundamental data and the “Center for Research in Security Prices (CRSP) monthly return tapes” were used to retrieve pricing data.

The CRSP is considered the “gold standard” for pricing data; we assume therefore that dividends were accounted for in the pricing data, though, no mention of dividends was made in the study. Presumably, the well respected “Value Line Investment Survey” was a reliable source of fundamental information.

It should be noted that no industries were excluded; at times financials (e.g. banks) are excluded from “value screens” due to the different structure of their balance sheets.

7. Human error

Given this study required the manual retrieval of fundamental data by the researcher, we think it prudent to assume some error may have occurred.

8. Journal rating/credibility[ii]

This study was not, to our knowledge, published in a top tier academic journal and therefore cannot be granted the “additional credibility” that may come with such publication.

**Reliability Assessment**: Given the short time period (< 10 years), small sample size post 1977 , arithmetic mean return calculation (i.e. “Raw Returns”) and the absence of a minimum market capitalization requirement we would not place reliance on the results of the study from a practitioner’s standpoint.

For the 24 months before and after a stock met the necessary criterion (i.e. Market Cap < NCAV) the author calculated the monthly:

- Raw Return
- Cumulative Raw Return
- Excess Return
- Percentage of Positive Excess Return
- Cumulative Excess Return

It will be important to emphasize the “excess returns” over the “raw returns” due to the sample of stocks being overwhelmingly concentrated in 1977. Focussing on excess returns will allow us to focus on the *relative* merits of stocks trading below NCAV. However, it should be noted that no definition of “excess returns” was identified in the study. Presumably it was the return above a broad market capitalization weighted index (such as the S&P 500) – an assumption we are forced to make to aid analysis.

Rather than reproduce the monthly statistics we calculate and summarise the key periods pre (t-24 to t-1) and post event (t0 to t24) along with our derivation of the cumulative market return (which was not reported in the original study):

It is stated that in *“the pre-event period (month -24 to month -1), the average cumulative raw return is 21 percent with a t-value of 1. 76”*. Reconciling this to the data we concur with the 21% return specified. Further, it is stated that the *“post-event cumulative raw return of 61.7 percent is significant at the 0.01 level (t = 5.93)”*. However, observing the return statistics it there appears to be a discrepancy; the post event cumulative raw return is actually 60.70% if month t0 is included – it makes sense to include month t0, otherwise the cumulative raw returns pre and post event do not reconcile. Such an adjustment does however leave us with a post event period of 25 months technically speaking.

Uncertainty over the exact pre and post event period aside, observing the cumulative return and cumulative excess return graphically reveals much:

Post event returns turn positive almost immediately. In the first year the cumulative raw return was 37.60% versus 22.60% for the market, an excess return of 15.00% (i.e. a 66.37% outperformance). In the second year the cumulative raw return was 23.10% versus 14.40% for the market, an excess return of 8.70% (i.e. a 60.41% outperformance). For the total two year post event period the cumulative raw return was 60.70% versus 37.00% for the market, an excess return of 23.70% (i.e. a 64.05% outperformance).

From the breakdown of the return statistics it appears that the first year post formation resulted in the bulk of return suggesting that a one year rebalancing period may prove to be return enhancing.

From April 1977 to December 1984 it appears as though firms trading below NCAV on the NYSE and AMEX demonstrated near perfect market timing capability – seemingly a magical time to be a deep value investor!

The study concludes with a foreshadowing… *“Future research is needed to determine if the NCAV rule continues its past trend and to determine if certain adjustments for firm* *size can change the profitability of this trading rule.”*

Indeed, the author along with Beni Lauterbach subsequently published a mathematically heavy research paper titled “Ben Graham's Net Current Asset Value Rule Revisited: The Size-Adjusted Returns”[iii]. The abstract states, *“The study demonstrates how size controls can alter the outlook of an investment strategy. The Ben Graham net current asset value rule provides excellent excess returns according to traditional performance measures. Size-adjustment procedures, however, reveal that its size adjusted excess return is approximately zero.” *We wonder what Graham would make of this assessment.

“An Empirical Analysis of Ben Graham's Net Current Asset Value Rule” by Joseph D Vu was an eight year study commencing in 1977 and concluding in 1984 that examined the returns to firms trading below Net Current Asset Value (NCAV) on the NYSE and AMEX exchanges.

Significantly the “raw returns” reported in the study were likely calculated as the arithmetic mean of returns; consequently, the reported returns would have overstated the actual returns achieved by an investor as measured by the more appropriate geometric mean. In addition, no minimum market capitalization was mandated for the firms examined, and therefore the study would have included even the smallest firms in the market. The inclusion of the very smallest firms may have biased the results as even when investing relatively modest sums, the securities of such firms are virtually untradeable.

**Notes:**
[i] Approximately equal to an inflation adjusted USD 219m in 2020 (https://www.usinflationcalculator.com/)
[ii] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/
[iii] Ben Graham's Net Current Asset Value Rule Revisited: The Size-Adjusted Returns, Beni Lauterbach and Joseph D. Vu, Quarterly Journal of Business and Economics, Vol. 32, No. 1 (Winter, 1993), pp. 82-108.

]]>

The focus of this post is the research paper “How the small investor can beat the market” by Joel M. Greenblatt, Richard Pzena[i] and Bruce L. Newberg published in the Journal of Portfolio Management (1981). The paper was their Master's thesis while studying at Wharton business school. The objective of the paper was to largely examine the performance of securities that were trading at or below “liquidation value” during the 6 year period from April 1972 to April 1978 in the US.

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the sub liquidation value method of investing.

Previous posts analysing the research related to securities trading at or below liquidation value include:

- An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”
- An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”
- An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London”
- Analyzing Deep Value in the Eurozone
- What Has Worked in Investing (Tweedy, Browne) – Examining Net Nets
- Examining Saudi Arabian Net Nets

• Valuation metric and portfolio construction parameters:
“Liquidating Value Per Share” = (Current Assets – Current Liabilities – Long Term Liabilities – Preferred Stock) / Number of shares outstanding
The authors measured the returns of four portfolios with their specific parameters identified below:
Portfolio 1:
Price/liquidation value: ≤ 1.0;
P/E: floating with bond yields;
Dividends: no dividend requirements.
Portfolio 2:
Price/liquidation value: ≤ 0.85;
PPE: floating with bond yields;
Dividends: no dividend requirements.
Portfolio 3:
Price/liquidation value: ≤ 1.0
P/E: ≤ 5.0;
Dividends: no dividend requirements.
Portfolio 4:
Price/liquidation value ≤ 0.85;
P/E: ≤ 5.0;
Dividends: no dividend requirements.
It is important to note that the sample they studied *“did not consider stocks that had shown a loss over the preceding 12 months”.*

• Weighting of holdings: Equal weight
• Formation date: *“we selected 15 segments of 4 months each over a six-year period.”*
This specification appears incorrect – the study extends for 6 years (i.e. 72 months) which would consist of *18* segments of 4 months (72 months / 4 = 18 segments).
It appears that the authors created “overlapping portfolios” which would have reduced the impact of “timing luck” i.e. choosing a formation that that coincided with a market bottom (top) thereby potentially biasing returns upwards (downwards).
• Sell strategy/Holding period: *“We sold a stock after a 100% gain or after 2 years, whichever came first.”*
It should be noted that no mention was made as to whether the capital was immediately reinvested when a stock generated a 100% gain. However, we assume that portion of the portfolio remained in cash until the end of the 2 year holding period.
• The stock universe:
*“We attempted to select a statistically significant and unbiased sample of stocks by compiling data on all firms listed in the Standard & Poor’s stock guide with market values over $3 million[ii] beginning with the letter A or B. This sample represented approximately 15% of all stocks listed in the guide, or 750 companies.”*
Presumably the Standard & Poor’s stock guide contained firms traded over the counter (OTC) as well as those listed on what was then known as the American Stock Exchange (AMEX) and the New York Stock Exchange (NYSE). For clarity it should also be noted that until the mid-1980’s the “NASDAQ” was synonymous with “OTC”[iii].

While there are numerous biases/errors that can be made when conducting studies/back tests, below we have analysed those we deem most likely to impact a study of this nature: 1. Survivorship bias No mention was made with regard to controlling for survivorship bias. However, given the authors used the Standard & Poor’s stock guide (as opposed to a computerized data base that might not include delisted stocks) the data source would logically be free from survivorship bias. 2. Look Ahead bias Controlling for look ahead bias was not specifically mentioned. We assume: • Portfolios formed in April were based on data from the preceding December • Portfolios formed in August were based on data from the preceding June • Portfolios formed in December were based on data from the preceding September Nonetheless, in the absence of an express statement addressing look ahead bias it remains a possibility. 3. Time period bias The study spans 6 years from April 1972 to April 1978. We classify the study period as an “inadequate/unreliable”. For reference: · < 10 years; inadequate/unreliable · 11 to 20 years; somewhat reliable · > 20 years; more reliable · > 40 years; most reliable

4. Minimum market capitalization

Some studies do not mandate a minimum market capitalization requirement for the stocks contained in the investment universe under examination. Consequently, the results in such studies can be unduly influenced by stocks that are, in practice, virtually untradeable, even when attempting to deploy relatively modest amounts of capital.

In this study a minimum market capitalization of “over $3 million” was required for a stock to be considered. However, even after adjusting for inflation stocks close to the stipulated $3 million cut off *may* still have been insufficiently liquid to enable trading to occur in a “reasonable manner”.[iv]

5. Human error Given this study was created in a rather manual and laborious manner where the researchers had to manually collect the necessary data, we think it prudent to assume some errors may have occurred. 6. Journal rating/credibility [v] While home to the research of many prominent academics and practitioners the Journal of Portfolio Management is not, to our knowledge, classified as a “top tier academic journal” and therefore cannot be granted the “additional credibility” that may come with such publication.

**Reliability Assessment:** Notwithstanding the short time period (6 years), potential inclusion of insufficiently liquid stocks and a small sample size the study appears to be generally reliable.

For nostalgic reference the original “Table 1” summarizing the portfolio returns is reproduced below, however, given the study was published in 1981 type face we have also re-entered the data and presented it in an easier to read format:

If you immediately and intuitively understand the data in Table 1 we refer you to the following:
*“I know you think you understand what you thought I said, but I'm not sure you realize that what you heard is not what I meant”* – attributed to Alan Greenspan
A few points with regard to the figures stipulated in Table 1:
1. Period Returns - the returns presented in the body of the table represent the “4-month percentage increase in the portfolio”. These returns do *not *reconcile to the “Annual Compound Return” specified at the foot of the table. Indeed, each 4 month return represents a separate portfolio (inferred from the general variance in the number of holdings from adjacent periods).
2. Annual Compound Returns – we assume the Annual Compound Returns were calculated by applying the stated Sell strategy/Holding period whereby the authors *“sold a stock after a 100% gain or after 2 years, whichever came first”*. It is also stated that returns were calculated by *“by assuming an equal dollar-weighted amount invested in each stock in the portfolio. Therefore, the percentage gain for the entire portfolio was merely an average of the percentage gain of the individual stocks”.* Critically, “Compounded Returns” reflect practitioner reality.
3. Sample Size - the authors *“use[d] only 15% of the S&P stock universe […] there appears to be an opportunity to obtain a diversified portfolio of between 50 and 350 stocks that meet the constraints of Portfolio 1. In following our constraints, incidentally, we were unable to find any “bargains” in the market peak period of April 1972 to April 1973.”* Nonetheless, the average portfolio contained 15 stocks and ranged from 0 to 52.
4. Dividends - the Annual Compound returns did *not *include dividends and are therefore *understated *(during the test period *“Dividends averaged between 3% and 4% annually”*).
5. Taxes, Commissions and slippage – *“we assumed commissions of 2.5% on purchase price plus a 2.5% bid/ask spread (the bid/ask spread was applied to the 60% of our stocks that were purchased over-the-counter), a 2.5% commission on selling price, and a 25% capital gains tax (over 90% of the stocks were held long enough to qualify for capital gains treatment).”* The commissions included are in line with the higher trading costs associated with the time period examined[vi]. Furthermore, the inclusion of returns including taxes, commissions and slippage is highly instructive for a practitioner as it addresses concerns over “potential limits to arbitrage”.
6. P/E floating with bond yields – *“We required a P/E corresponding to twice the prevailing triple A yield in each period (e.g. triple A yield = 8%; required PPE equal or below the reciprocal of 16%, or 6.25).”* The Triple A bond yield during the period studied it ranged from approximately 7 to 9[vii] implying a requirement for a P/E below 5.5 to 7. This means that the P/E requirement of ≤ 5 for Portfolio 3 and 4 was less than the maximum floating P/E allowed for Portfolio 1 and 2.
Having laid out the above we are better placed to analyze the results, bearing in mind the relatively small sample size, notwithstanding that the authors “attempted to select a statistically significant and unbiased sample of stocks”.
The highest returning portfolio was Portfolio 4 (42.2%) which required stocks in the portfolio to possess the lowest valuation in terms of both liquidation value (≤ 0.85) and P/E ratio (≤ 5.0).
Interestingly, Portfolio 3 (32.2%) returned more than Portfolio 2 (27.1%) despite its allowance for a higher liquidation cut off (≤ 1.0 vs. ≤ 0.85 for Portfolio 2) but a lower P/E threshold (≤ 5.0 vs ≤ ~5.5 to 7 for Portfolio 2).
Portfolio 1 which allowed for the highest valuation in terms of liquidation value (≤ 1.0) and P/E (≤ ~5.5 to 7) generated the “lowest” return (20.0%).
Overall, the results imply valuation drives futures returns, however, the merit of liquidation value vs the P/E ratio to explain future returns is less clear.
In terms of relative returns all portfolios greatly outperformed the “OTC” and “Value Line” portfolios which generated an Annual Compound Return (before taxes and commissions) of 1.3% and -0.3% respectively. Significantly, the outperformance of all portfolios survived taxes, commissions and slippage. However, all returns during the period were severely impacted by the ravages of inflation which averaged approximately 7.5%[viii] per annum (geometric mean) from 1972 to 1976.

Despite the particularly short test period (6 years), limited portfolio holdings (average of 15), potential inclusion of stocks with insufficient liquidity to enable trading and the presentation of convoluted and seemingly irrelevant return data, “How the small investor can beat the market” still possesses some utility. It demonstrated that the Annual Compound Returns for firms with positive earnings trading at or below liquidation value, combined with a low price to earnings ratio greatly outperformed OTC and Value Line firms during the examination period (April 1972 to April 1978) even after taking into account taxes, commissions and slippage.

While the examination period was simply too short to provide *definitive* guidance as to the efficacy of investing in firms trading at or below liquidation value, the study nonetheless represents a small, though not untainted piece, in the larger puzzle of determining what we “really know” about returns of such firms from an empirical standpoint.

**Notes:**

[i] MarketFox Interview with Rich Pzena in which he briefly discusses the study (https://i3-invest.com/podcasts/i3-podcast-marketfox-interview-with-rich-pzena/)

[ii] Approximately equal to an inflation adjusted USD 18m in 2020 (https://www.usinflationcalculator.com/)

[iii] https://en.wikipedia.org/wiki/Nasdaq

[iv] James O'Shaughnessy, What Works on Wall Street: The Classic Guide to the Best-Performing Investment Strategies of All Time, 4th ed. Chapter 5 (New York: McGraw-Hill, 2011).

[v] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

[vi] A Century Of Stock Market Liquidity And Trading Costs, Charles M. Jones, Graduate School of Business Columbia University, 2002 (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=313681)

[vii] https://fred.stlouisfed.org/series/AAA

]]>

This is our sixth post in our series on “net nets” having previously published:

- An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”
- An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”
- An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London”
- Analyzing Deep Value in the Eurozone
- What Has Worked in Investing – Examining Net Nets

The focus of this post is the research paper “Emerging Markets: Evaluating Graham’s Stock Selection Criteria on Portfolio Return in Saudi Arabia Stock Market” by Nadisah Zakaria and Fariza Hashim. The objective of the paper was to examine the performance of securities that were trading at greater than 1.5 of Net Current Asset Value (NCAV)/Market Value (MV) (i.e. less than 2/3 of NCAV) during the 15 year period from 2000 to 2011 on Tadawul (i.e. the Saudi Arabian Stock Exchange).

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the “net net” method of investing.

- Valuation metric:

*“In complying with the primary criteria of Graham’s approach, companies with per share NCAV greather than 1.5 times the current asset share price were identified. Investors should select NCAV/MV shares with a margin of safety: At a price no more than two thirds of the company’s NCAV.”*

*“In calculating the NCAV of the companies listed, all data regarding the companies’ current assets, current liabilities, long-term debt, and preferred stock were downloaded from balance sheet entries on Datastream (Xiao and Arnold, 2008).”*

From the above we deduce that the authors calculated the NCAV per share as follows:

Net Current Asset Value per share = (Current Assets – (Current Liabilities + Long Term Debt + Preferred Shares))/Common Shares Outstanding

The number of firms meeting the “primary criteria” were identified in Table 1, replicated below:

What we observe is that the sample size is small and grows progressively smaller throughout the study period, despite the Global Financial Crisis (2007-2009).

However, the selection criteria does not end there.

*“Those companies that fulfilled the primary criteria were further investigated to ensure they satisfy the secondary criteria, which are summarized in Table 2. It is important for the companies to comply with the two levels of criteria to enable them to be further analyzed.”*

Now, the authors didn’t disclose how many firms qualified for investment based on the primary *and* secondary criteria but logically we know it can only be less than or equal to the number of firms that met the primary criteria as stipulated in Table 1.

With regard to “Satisfactory earnings” (refer Table 2) – to find a group of net nets with compounding earnings over such an extended time frame would be quite a feat. Companies don’t ordinarily trade below a conservative liquidation value while simultaneously compounding earnings. Indeed, one may wish to examine the financial statements forensically in such a case. Furthermore, the authors state that *“The examining period for this study is from January 2000 until December 2014. This is due to the fact that prior to the year 2000, company data is not yet available in the Saudi Arabia stock exchange.” *So, it isn’t clear to us how, or what source was used to retrieve historical data of such length to make the required assessment in accordance with this criterion.

It is stated that “Companies should have price to P/B ratio below 1”, however, this criterion is redundant because they would be subsumed by firms meeting the primary criteria.

It should be noted that applying *just* the primary criteria resulted in fewer than 10 firms being eligible for investment from investment from 2007 to 2011. Applying the secondary criteria could have only further reduced the numbers of firms examined as alluded to previously. Overall, the sample size appears to be particularly small in this study.

- Weighting: Equal weight and Value weight

- Formation date:
*“the study created an annual portfolio share in the month of July.”*

- Holding period:
*“The buy-and-hold portfolios are constructed for 1, 2 and 3 years, with the first formation in July 2000 and the last formation in July 2011.”*

- The stock universe:

*“As of 31**st **December 2011, a total of 160 companies were listed on Tadawul with a market capitalization of USD385.3 million. Interestingly, the Saudi Arabian share market is the largest in the region, accounting for about 50% of the six AGCC markets: Bahrain, Kuwait, Oman, Qatar, and UAE. Indeed, more than 80% of all shares trading in terms of value take place in Saudi Arabia.”*

By US standards the entire market capitalization of the Saudi Arabian share market was equivalent to a microcap or two! Furthermore, it should be noted that “In June 2015, Saudi Arabia’s stock exchange experienced a major change when it started to open to foreign participation” i.e. foreign investors faced a genuine limit to arbitrage during the test period.

While there are numerous biases/errors that can be made when conducting studies/back tests, below we have analysed those we deem most likely to impact a study of this nature:

1. Survivorship bias

No mention was made with regard to controlling for “survivorship bias”.

The data source may have suffered from survivorship bias. However, given the study parallels the commencement of the exchange itself, the impact of survivorship bias, if any, may have been smaller than would otherwise be expected.

2. Look Ahead bias

*“Following Xiao and Arnold (2008), the study created an annual portfolio share in the month of July. Companies are required to have data for NCAV in December of t-1. This enabled the researcher to observe at least one return in the post-formation period.”*

Based on that mentioned above the researchers appear to have avoided look ahead bias.

3. Time period bias

The study spans 15 years and we classify this as a “somewhat reliable” period.

For reference:

· < 10 years; inadequate/unreliable

· 11 to 20 years; somewhat reliable

· > 20 years; more reliable

· > 40 years; most reliable

4. Data source and treatment

*“In calculating the NCAV of the companies listed, all data regarding the companies’ current assets, current liabilities, long-term debt, and preferred stock were downloaded from balance sheet entries on Datastream (Xiao and Arnold, 2008). Companies that are listed as financial sectors are excluded from this analysis. Returns for each company including dividends were adjusted for changes in stock split, rights issues and share repurchases were obtained from Thompson Reuters Datastream, one of the major and authoritative financial information providers.”*

While we do not have any specific knowledge as to the reliability of the data sets used, from the above extract it is clear the researchers took specific measures to ensure their data was reliable and that the companies, markets and sectors examined were also appropriate.

5. Minimum market capitalization

No minimum market capitalization cut off for the securities examined in this study was mandated. Consequently, the inclusion of the smallest firms could have unduly influenced results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable.

6. Human error

There is nothing specified in the research methodology that would make us believe this study is at greater risk of suffering from human error.

7. Journal rating/credibility[i]

**Reliability Assessment:** Due to the relatively short time period (15 years), sample size concerns and the absence of a minimum market capitalization requirement for the securities examined the study does not appear to be reliable from a practitioner’s viewpoint.

Table 3 summarizing the relevant returns is reproduced below:

*“Panel A shows the results of average market-adjusted BHAR on the Saudi Arabia stock exchange (Tadawul) in a period of 12, 24 and 36 months. The stock portfolios analyzed are those that comply with Graham’s primary and secondary stock selection criteria. The results were analyzed separately based on the two indices; EWI and VWI, The results of the t-test for the stock portfolios from the EWI index against BHAR was 2.9 (P = 0.01) and BHAR was 20.17% for the 1-year period; 4.87 (P = 0.01) and BHAR was 46.7% for the 2-year period; and 6.1 (P = 0.01) and BHAR was 83.47% for the 3-year period. These results indicate that the longer the investors hold the portfolios, the higher the abnormal return. Nevertheless, the results of stock portfolios from VWI demonstrated a slightly different scenario. The t-test result for the stock portfolios from the VWI against BHAR was 1.34 (not significant at P = 0.1) and BHAR was 8.62% for the 1-year period; 3.28 (P = 0.01) and BHAR was 27.69% for the 2-year period; and 3.31 (P = 0.01) and BHAR was 49.02% for the 3-year period. These results indicated that investors gained abnormal returns only after a year of investment, which signifies that the abnormal return was secured but requires a longer period of holding the portfolios.”*

*“The findings above illustrate that the NCAV/MV portfolios that were measured against the market benchmark of SAS-EWI significantly performed exceeding the expectation on average by +83.47% in the 3-year holding period. Correspondingly, when the NCAV/MV portfolios were measured against the market benchmark of SAS-VWI, they also demonstrated a positive and significant market-adjusted BHAR of +49.02% over the 3-year period, though the percentage of return was marginally lower. This situation indicated that smaller companies outperformed the larger companies on the Saudi Arabia Stock Market during the period of study.”*

As we have done throughout this series on net nets we will examine how the returns were calculated and what they actually represent. In this study the authors explained the methodology they used to calculate returns:

*“In analyzing the return, we focused on abnormal return, which is defined in this context as the difference between the actual return and the expected return of individual stocks in the portfolios. Despite Lyon et al. (1999. p. 198) reminding us regarding the use of buy-and-hold abnormal returns (BHAR), extensive literature supports the use of the BHAR method as it copes better with the effect of compounding than does cumulative abnormal return (CAR) (Ritter, 1991; Barber and Lyon, 1997), Fama (1998) also argued that compounding short-term returns to obtain long-term BHAR better captures long-term investor experience. In fact, using merely the average abnormal returns used in the CAR approach does not accurately measure returns to investors over the long-run period. In modern event studies, the most commonly accepted methodology is the BHAR approach. Therefore, this research adopts this method to evaluate the share return performance of NCAV portfolios over the long-run period. Ritter (1991) asserts that in minimizing the problem related to measuring portfolios, benchmarking those portfolios is vital. Therefore, we employ two market indices: Equal-weight index (EWI) and value weight index (VWI).”*

Right, that is an extensive explanation, but what *are* “*buy-and-hold abnormal returns (BHAR)”? *In essence, in this instance, the BHAR represents the *geometric mean return less the market return*.

So, while equally weighting and holding 12 months produced a 20.17% BHAR the next question that comes to mind is, “what was the equally weighted market return that was outperformed by 20.17%”? Unfortunately, this was not reported.

It is worth reinforcing that value weighting (“VWI”) resulted in a materially lower BHAR implying that the smaller companies were driving the “abnormal returns”.

“Emerging Markets: Evaluating Graham’s Stock Selection Criteria on Portfolio Return in Saudi Arabia Stock Market” suffered from time period bias (15 years) and the sample size examined was particularly small. Significantly, no minimum market capitalization was specified for the securities examined. Consequently, the inclusion of all firms may have unduly influenced the results as even when investing relatively modest sums the securities of the very smallest firms are virtually untradeable. Furthermore, the market addressed possessed a capitalization of less than USD 400m raising further concerns with regard to liquidity. Also noteworthy is that regulatory restrictions during the study period would have prevented foreign investors from being able to access the potential opportunities.

**Notes:**
[i] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

Celebrated value investors Tweedy, Browne Company LLC published a booklet in 2009 that contained over 50 studies titled “What Has Worked in Investing: Studies of Investment Approaches and Characteristics Associated with Exceptional Returns”. Of particular interest to us was their inclusion of a study on the returns achieved by “net nets”.

The objective of the original study was to examine the performance of securities that were selling at 66% or less of net current asset value from April 1970 to April 1981, thereby creating a 12 year study period.

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the “net net” method of investing.

Other posts in our series on net nets include:

- An Analysis of “Benjamin Graham’s Net Current Asset Values: A Performance Update”
- An Analysis of “Graham’s Net-Nets: Outdated or Outstanding?”
- An Analysis of “Testing Benjamin Graham’s Net Current Asset Value Strategy in London”
- Analyzing Deep Value in the Eurozone

- Valuation metric

While a specific formula was not mentioned on page 5 where the study commences, earlier in the publication the authors mention *“The net current assets investment selection criterion calls for the purchase of stocks which are priced at 66% or less of a company's underlying current assets (cash, receivables and inventory) net of all liabilities and claims senior to a company’s common stock (current liabilities, long-term debt, preferred stock, unfunded pension liabilities).”* (p.1)

We therefore assume this is the formula applied, with the exception of the incorporation of “unfunded pension liabilities” which would have required a review of the annual reports for each company to identify – a task unlikely to have been undertaken for the purposes of this study.

- Weighting:
*Not specified*

- Purchase/rebalance date:
*“April 30 in each of 1970 through 1981”*

While rebalancing on the same date each year is consistent it nonetheless exposes the results to “timing luck”. (To limit the impact of timing, researchers often create portfolios each month using overlapping periods).

- Holding/rebalancing period:
*“investment returns for all stocks were computed for 6 months, 1 year, 2 years and 3 years after each selection date.”*

While there are numerous biases/errors that can be made when conducting studies/back tests, below we have analysed those we deem most likely to impact a study of this nature:

1. Survivorship bias

*“All 7,000 public companies in the Compustat database, including the Research File of companies which had been acquired, merged or declared bankrupt subsequent to an assumed historical selection date”*

The data source appears to be free from “survivorship bias”.

2. Look Ahead bias

Controlling for “look ahead” bias was not specially mentioned, however, given the rebalance date of 30 April it is likely that the study used financial statements ending 31 December, thereby avoiding look ahead bias.

3. Time period bias

The study spans 12 years and we classify this as a “somewhat reliable” period.

For reference:

· < 10 years; inadequate/unreliable

· 11 to 20 years; somewhat reliable

· > 20 years; more reliable

· > 40 years; most reliable

4. Data source and treatment

*“All 7,000 public companies in the Compustat database”.*

Compustat is a reputable database.

It should be noted that the study included firms with a *“market capitalization of at least $1 million” *i.e. “nanocap” firms formed part of the investment universe examined. However, no mention of the number of firms that qualified as investment candidates was mentioned. We understand that the Compustat database includes securities listed on the NYSE, AMEX and NASDAQ exchanges.

Interestingly, the Oppenheimer study was conducted over a nearly identical time period and he did specify the number of securities in his study.

Table I from Benjamin Graham’s Net Current Asset Values: A Performance Update by Henry R. Oppenheimer is reproduced below:

For clarity it should also be noted that until the mid-1980’s the “NASDAQ” was synonymous with “OTC”[i]. Therefore, it is likely that the universe of stocks examined was nearly identical to that examined in the Oppenheimer study.

5. Human error

6. Journal rating/credibility[ii]

**Reliability Assessment:** Aside from the relatively short time period, 12 years, the study *appears* to be reliable.

Table 4 from What Has Worked in Investing is reproduced below:

The “Average Return” is almost certainly the arithmetic mean return which *overstates* the actual returns attainable by an investor in practice. We reached this conclusion as a number of other studies published in the booklet expressly state that returns were “compounded”, whereas in this case no such specification was made. Furthermore, using historical data we found that the specified 1 Year S&P500 return of 8.5% reconciles almost perfectly to the S&P500 total return arithmetic mean return calculated using *calendar* year returns[iii]. Oddly, the 1 Year S&P500 return presented against “Stock Selection Criteria” “66% of net current asset value” is different to all other portfolios (i.e. 9.1% vs 8.5%). Presumably this is because a different time period was used for these portfolios, but we did not note any explanation for this difference in the study.

Further, it is stated that *“One million dollars invested on April 30, 1970 and rolled over at each subsequent April 30 into the stocks selling at less than 30% of book value would have increased to $23,298,000 on April 30, 1982. One million dollars invested in the S&P 500 on April 30, 1970 would have been worth $2,662,000 on April 30, 1982.”* i.e. $1m compounded at 30.0% for 12 years.

An otherwise astute reader may take the stipulated ending value of $23,298,000, starting capital of $1,000,000 and the 12 year time horizon to “confirm” the *apparent* geometric mean of 30.0% (i.e. r = (23,298,000/1,000,000)^(1/12) -1). While seemingly astute it is nonetheless inconsistent with what was *actually* reported – the arithmetic mean. By ostensibly attempting to be helpful and express the returns in dollar terms, the authors may have inadvertently delivered something of a “one-two knockout” to the reader; one, use a method that overstates returns (arithmetic mean) and two, utilise an example *implying* the result represents the compound annual growth rate to further exacerbate the issue!

This is another reminder to actively go in search of disconfirming evidence and question everything, irrespective of its source.

“The important thing is not to stop questioning.” (Albert Einstein)

What Has Worked in Investing is a wonderful resource; nonetheless it is important to read each study contained therein analytically and be cognisant of the different methodologies used in calculating and presenting the reported returns.

For a number of reasons, we would *not* place any great reliance on the results reported in this particular study. Firstly, it covered only 12 years, a relatively short period when trying to ascertain the efficacy of an investment strategy. Secondly, and significantly, it invested in companies with market capitalizations as low as one million USD, which, even by today’s inflation adjusted standards would likely see many investors face liquidity constraints and consequently a “limit to arbitrage”. Finally, and critically, the “Average Return” reported in the study was calculated as the *,arithmetic* mean of returns, which would have *,overstated* the actual returns achievable by an investor as measured by the more appropriate geometric mean.

**Notes:**

[i] https://en.wikipedia.org/wiki/Nasdaq

[ii] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

[iii] http://www.stern.nyu.edu/~adamodar/pc/datasets/histretSP.xls

]]>This is our fourth post in our series on “net nets” having previously analyzed “Benjamin Graham’s Net Current Asset Values: A Performance Update” by Henry R. Oppenheimer, “Graham’s Net-Nets: Outdated or Outstanding?” by James Montier and “Testing Benjamin Graham's Net Current Asset Value Strategy in London” by Ying Xiao and Glen C. Arnold.

The focus of this post is to analyze the research paper, “Studying Different Systematic Value Investing Strategies on the Eurozone Market” conducted by investment practitioners Philip Vanstraceele and Luc Allaeys.

The paper examined the performance of a variety of value investing screens during the 10 year period from 1999 to 2009 for stocks listed in the “Eurozone”. Of interest to us, the paper examined the performance of stocks identified by a “Net Current Asset Value” (NCAV) screen.

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the Net Current Asset Value (i.e. “net net”) method of investing. While the authors examined four value investing screens, we focus our analysis on the Net Current Asset Value method.

- Valuation metric:

*“NCA= Current Assets (cash, inventories and accounts receivable) – Total Liabilities*

*This strategy calls for buying stocks trading at 2/3 or less of their NCAV.*

*In our methodology the NCAV-ratio should be greater than 1,33, and we define the ratio as Net Current Assets Value / Market Value.*

*This ratio is used to find companies that are trading below their net current assets value.”*

For clarity, it should be noted that a ratio of 1.33 of Net Current Asset Value/Market Value equals 75% of NCAV (not 2/3 i.e. 67% of NCAV more typically associated with the strategy as mentioned by the authors). In addition, there is no mention of the treatment of preferred shares, therefore we assume they were not taken into consideration.

- Weighting: “
*equally weighted basis”*

- Holding/rebalancing period: “
*buy and hold strategy for 1 year… and rebalanced the portfolios once a year at exactly the same time” i.e. on 13 June.*

While rebalancing on the same date each year is consistent it nonetheless exposes the results to “timing luck”. For example, if portfolios were rebalanced in March it is highly likely that the returns would appear materially lower given that most equity markets reached their trough in March 2009. In contrast, if portfolios were rebalanced in October, it is highly likely that the returns would appear materially higher given that most equity markets saw appreciable recovery between March 2009 and October 2009.

The point being, consistency, while desirable, cannot alone be equated with reliability.

- Market capitalization and the number of holdings:

The authors examined 20 stock and 50 stock portfolios and did so at various minimum market capitalization cut offs.

The portfolios examined are identified below:

While there are numerous biases/errors that can be made when conducting studies, below we have analyzed those we deem most likely to impact a study of this nature:

1. Survivorship bias

*“Most studies don’t include companies that went bankrupt or that were taken over by others. In our test these company’s [sic] were not excluded.”*

Based on the above the data source appears to be free from survivorship bias.

2. Look Ahead bias

*“When you are using data for your stock ranking that was not available at the moment of portfolio formation, your results will suffer from look-ahead bias. This biases results upwards.*

*We worked with accounting data from the prior fiscal year and waited 6 months to form our portfolios and actual trading.”*

Given the “six-month lag” the study appears to be free from look ahead bias.

3. Time period bias

The study spans 10 years and we classify this as an “inadequate/unreliable” period.[i]

However, given the period incorporated the Great Financial Crisis that saw many markets drawdown 50% we do accept the premise put forth by the authors:

*“All things considered, the last decade has been quite turbulent for the stock markets all over the globe. We had one bull market and two major crashes (a “luxury” for back-testers).”*

Nonetheless, an investor would be well advised to refrain from making capital allocation decisions based solely on a 10 year back test.

4. Data source and treatment

- Data source:
*“Thomson DataStream”* - Excluded industries:
*“We excluded Financial- and Insurance companies from our screeners!!”* - Universe:

*“In the Euro Monetary zone there are +/- 4.080 companies with a primary quote on the stock market. If we exclude the financials and insurers are excluded +/- 3.400 companies remain.”*

We assume that the abovementioned represents the stock universe available for examination “from the end of 1998” and that new listings were incorporated after the commencement of the test period[ii].

- Dividends:

*“We back tested the different strategies and calculated the price-index “PI” (excluding dividends)”.*

Excluding dividends when measuring returns would have certainly understated the return achieved. Furthermore, for firms that were able to sustain the quantum of their dividends during the Global Financial Crisis, they would have, on the balance of probability offered a particularly high dividend yield. Thus, not accounting the dividend component would have, in our view materially impacted the reliability of the back test. Confusingly, in a subsequent set of tests the authors state: “Here we included the dividends”. So, when examining the results, we need to be mindful as to which set of tests we are analyzing given some accounted for dividends and some did not.

- Eurozone:

The “Eurozone” consists of European Union (EU) member states which have adopted the euro (€) as their common currency and sole legal tender. The authors did not expressly state which countries/markets formed part of the Eurozone during the period studied. Complications arise given some member states adopted the Euro *during* the study period. Were they included following their adoption of the Euro or were only markets that had adopted the Euro at the commencement of the study included? While not mentioned in the study our correspondence with the author confirmed that countries that adopted during the period were included. That means Greece, in consequence of their adoption of the Euro on 1 January 2001 would have formed part of the study. While seemingly trivial had they been excluded it may have had a material impact on the results given the performance of their stock market during the period examined. Furthermore, it is during major drawdowns that net nets are more likely to be available and therefore their potential exclusion could have had a material impact on results.

For completeness we have included the list of countries that had adopted the Euro at the commencement of, and during the study period (13 June 1999 to 13 June 2009) in the endnotes[iii].

5. Citation Error

It is stated that:

*“One research study, covering the years 1970 through 1983 showed that portfolios picked at the beginning of each year, and held for one year, returned 29,4%, on average, over the 13 year period, compared to 11,5% for the S&P 500 Index.”*

The authors are almost certainly referring to the study “Benjamin Graham’s Net Current Asset Values: A Performance Update” by Henry R. Oppenheimer. While minor, the comparative benchmark to which they are likely referring to was actually the “NYSE-AMEX Index” as opposed to the stipulated “S&P 500 Index”.

6. Journal rating/credibility[iv]

**Initial Reliability Assessment:** Despite a very concerted effort to ensure the study did not suffer from major biases the relatively short test period (10 years) leads us to conclude that in isolation the results cannot be fully relied on, however, they could provide the basis for further research.

Let the games begin!

So as identified above the authors test net nets across different market capitalizations and number of holdings. For example, they examined net net portfolios with minimum market capitalization of EUR 25m and 20 companies (“MV25 C20”) and net net portfolios with a minimum market capitalization of EUR 5bn and 20 companies (“MV5.000 C20”).

Wait, what?!

Is that to say in *each and every year *the authors were able to identify *at least* 20 net nets (per their specified criteria – “*NCAV-ratio should be greater than 1,33”)* across the entire spectrum of market capitalizations tested? Well, we could believe that is possible for the portfolios with a minimum market capitalization of EUR 25m, but to find *at least* 20 net nets with a market capitalization of EUR 5bn for ten consecutive years?! That seems utterly implausible! And so, it is…

In fact, what was *actually* tested were those firms that traded at the highest *relative *“ratio of Net Current Assets Value / Market Value” (MV) **not** what was specified in the methodology of the study (refer section “Key Methodology”)! Significantly, there was no way to have definitively reconciled this incongruity other than by asking the authors.

Now, having identified that this study does **not** examine what it stated (i.e. firms trading at a NCAV ratio of greater than 1.33, or said in more familiar terms, 75% of NCAV), but rather firms trading at the highest relative ratio of NCAV / MV (i.e. the “cheapest) we proceed with a review of the returns achieved.

Significantly, and thankfully, the authors examined the returns by calculating the Compound Annual Growth Rate (CAGR) (as opposed to the arithmetic average return). Out of interest, the arithmetic average return for the DJ Euro Stoxx was -0.24% which, prima facie doesn’t sound calamitous given the literal calamity that transpired in financial markets during the period examined. However, the truth of the matter is (gross of fees) an index tracking investor compounded at -3.13% (including dividends)[v] and therefore lost a cumulative 27.22% for the 10 years ended 13 June 2009!

We have restated the returns as reported in the study with regard to portfolios constructed using the NCAV/MV metric below:

For simplicity and brevity, we will review the returns of the relatively cheapest stocks as measured by NCAV/MV for three groups:

1. MV25 C20 (minimum market capitalization of EUR 25m and 20 companies):

The NCAV MV25 C20 generated a CAGR of 13.44% vs the DJ Euro Stoxx -3.13%, an outperformance of 16.57%. Certainly, generating a double digit return while the market return was negative would have been heartening!

2. MV250 C20 (minimum market capitalization of EUR 250m and 20 companies):

The NCAV MV250 C20 generated a CAGR of 11.06% vs the DJ Euro Stoxx -3.13% an outperformance of 14.19%. Significantly at a minimum market capitalization of EUR 250m some institutional investors would be able to access the outperformance.

3. MV5.000 C20 (minimum market capitalization of EUR 5bn and 20 companies):

The NCAV MV5000 C20 generated a CAGR of -1.55% vs the DJ Euro Stoxx -3.13% an outperformance of 1.58%. While the outperformance of 1.58% for the largest capitalization stocks examined appears trivial, it is important to note that the cumulative consequence was a loss of 11.45% (100 - 85.55) as opposed to 27.22% (100 - 72.78); a little testament to the power of compounding!

Observing the returns over time one can’t help but notice that returns between the largest group examined (“MV 5000 C20) along with the market (“DJ Euro Stoxx) appear to have varied materially from those attained by the portfolios of smaller firms.

Below we chart the annual return of four (of the 13) portfolios tested along with the annual market returns[vi].

While oftentimes directionally consistent, the variation in the magnitude of returns over the time period studied is noteworthy.

Calculating the correlation coefficient between the returns of the portfolios charted above demonstrates that the returns of smaller stocks took an entirely different path to those of larger stocks and the overall market.

Lastly, we chart below the compound annual growth rate of the 20 stock portfolios (i.e. “C20”) along with that of the market:

From the chart above it would appear that as the minimum market capitalization was increased returns fell in a consistent, though not perfect sequence. A regression analysis was not conducted for various “factors”. However, a reasonable hypothesis might attribute the pattern as consistent with the “size factor”, along with the probability that as the investment universe shrunk (by increasing the market capitalization requirement) it contained firms with a lower relative NCAV/MV ratio (i.e. more expensive).

Now, let’s not forget, these tests *excluded *dividends when it came to the quantification of returns to the NCAV portfolios, however the DJ Euro Stoxx Index *included* dividends[vii]. Including dividends, it would be reasonable to assume that it would have resulted in a material increase in returns to the NCAV portfolios *not *accounted for in the results displayed.

So, there it is, the cheapest firms as identified as those with the highest *relative* NCAV/MV ratio produced market beating return from 1999 to 2009 in the Eurozone across a full spectrum of market capitalizations - case closed.

Well, not so fast…

The authors also split their stock universe up into deciles (with various minimum market capitalization requirements) and presented the returns (this time *including* dividends).

The authors make a particularly thought provoking comment in their conclusion:

*“Using the NCAV and splitting it up in deciles sometimes gives you strange results but you have to consider that using NCAV doesn’t always reflect “how cheap” a company is.”*

We analyse this comment in further detail.

The minimum market capitalizations for the decile tests are EUR 10m, EUR 100m and EUR 500m respectively.

To get a sense of the size of the universe, and therefore the number of firms in each decile, we provide an extract from the study identifying the size (and market capitalization) of the investment universe examined:

While the market cap classifications above don’t neatly match those used in the decile analyses, we can deduce that the minimum numbers of firms contained in each decile for the 10m, 100m and 500m minimum is not less than 190, 71 and 38 firms respectively[viii].

Given that we have a MV100 C20 portfolio and a MV500 C20 portfolio to examine we will focus on the deciles returns with the same respective minimums i.e. 100m and 500m - this ensures the market capitalizations of stocks match when conducting our analysis.

Below we restate the returns of decile 1 (D1)[ix]and decile 2 (D2) (minimum market capitalization 100m) alongside those of the MV100 C20 portfolio for comparative purposes:

Recall MV100 C20 consists of the 20 firms that possessed the highest ratio of NCAV/MV i.e. the cheapest stocks. Therefore those 20 firms would be subsumed within D1. While the cheapest 20 firms generated a CAGR of 10.48% the cheapest decile, which contained those firms, generated a CAGR of -2.18%. Yet, D2 generated a CAGR of 11.93%.

The above implies a lack of robustness to the hypothesis that cheapness, as measured by the NCAV/MV is the driver of returns, for if it was, we would expect a consistent diminution of returns as the NCAV/MV decreased (i.e. more expensive), as opposed to the inconsistent sequence identified above.

Furthermore, it should be noted that the 10.48% CAGR achieved by the MV100 C20 portfolio *excluded* dividends, whereas the -2.18 CAGR of D1 *included* dividends, further confounding the results.

A chart of annual returns for D1, D2 and MV100 C20 is provided below:

Continuing in the same vein below we restate the returns of D1 and D2 (minimum market capitalization 500m) alongside those of the MV500 C20 portfolio for comparative purposes:

The results with a minimum market capitalization of 500m are very similar to those found for a minimum market capitalization of 100m. Specifically, MV500 C20 consists of the 20 firms that possessed the highest ratio of NCAV/MV i.e. the cheapest stocks. Therefore those 20 firms would be subsumed within D1. While the cheapest 20 firms generated a CAGR of 7.23% the cheapest decile which contained those stocks generated a CAGR of -6.89% (a 51% cumulative loss!). Yet, D2 generated a CAGR of 7.23%.

Again, the above implies a lack of robustness to the hypothesis that cheapness, as measured by the NCAV/MV is the driver of returns, for if it was, we would expect a consistent diminution of returns as the NCAV/MV decreased (i.e. more expensive), as opposed to the inconsistent sequence identified above.

A chart of annual returns for D1, D2 and MV500 C20 is provided below:

The authors stated that they were examining the returns to stocks that traded at a minimum NCAV/MV ratio of 1.33 (i.e. 75% of NCAV), however we subsequently confirmed that what was *actually* tested were stocks that traded at the cheapest *relative* valuation as measured by NCAV/MV with no constraints to the absolute value of the ratio itself.

Initially, the CAGR achieved by portfolios across a wide spectrum of market capitalizations seemed to definitively indicate the firms with the highest NCAV/MV (i.e. the cheapest) reliably and consistently outperformed the market as measured by the DJ Euro Stoxx. However, the robustness of the results was brought into question when we analysed the returns to the decile portfolios which revealed an inconsistency which could not be explained.

The lack of robustness in returns notwithstanding, and despite a very concerted effort to ensure the study did not suffer from major biases the relatively short test period (10 years) would have always led us to conclude that, in isolation, the results could not be fully relied on. Nonetheless, while uncertainty as to the efficacy of the NCAV/MV as a standalone metric remains, the results also imply that it is also worthy of further examination.

Sometimes, to see further one must dig deeper.

**Notes:**

[i] For reference:·

- < 10 years; inadequate/unreliable
- 11 to 20 years; somewhat reliable·
- > 20 years; more reliable·
- > 40 years; most reliable

[ii] Confirmed by author.

[iii] Eurozone countries (https://en.wikipedia.org/wiki/Euro): Monaco, San Marino and Vatican City do not have a stock exchange.

[iv] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

[v] The DJ Euro Stoxx index *included* dividends (confirmed by author).

[vi] We opted not to chart all 13 portfolios to ensure the chart was readable.

[vii] While the NCAV portfolios *excluded* dividends the DJ Euro Stoxx index *included *dividends (confirmed by author).

[viii] 10m: 77+308+334+1189 = 1908, 1,908/10 = 190.8. 100m: 77+308+334 = 719; 719/10 = 71.9. 500M: 77+308 = 385/10 = 38.5.

[ix] D1 contains the firms highest NCAV/MV ratio i.e. the cheapest.

]]>This is our third post in our series on “net nets” having previously analyzed “Benjamin Graham’s Net Current Asset Values: A Performance Update” by Henry R. Oppenheimer and “Graham’s Net-Nets: Outdated or Outstanding?” by James Montier.

The focus of this post is the research paper “Testing Benjamin Graham’s Net Current Asset Value Strategy in London” by Ying Xiao and Glen C. Arnold. The objective of the paper was to examine the performance of securities that were trading at greater than 1.5 of Net Current Asset Value (NCAV)/Market Value (MV) (i.e. less than 2/3 of NCAV) during the 26 year period from 1980 to 2005 on the London Stock Exchange. In addition, the authors sought to examine whether the “excess returns” of such stocks could be explained by “risk”, the size effect, the Capital Asset Pricing Model (CAPM), the Fama-French three Factor Model (FF3M) or investor irrationality.

Our objective was to analyze the study itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the “net net” method of investing.

- Valuation metric:

The authors stipulate how *Benjamin Graham* quantified NCAV per share:

*“Graham’s NCAV/MV strategy calls for the purchase of stocks at a price 2/3 or less of the NCAV. Per share NCAV, as defined by Graham (Graham and Dodd (1934), Graham (1976)), is the balance sheet current assets minus all the firm's (current and long-term) liabilities divided by the number of shares outstanding.”*

However, it was not obvious to us that the authors replicated the abovementioned formula exactly.

They mention that:

*“In order to calculate NCAV, current assets, current liabilities, long-term debt and preferred stock are downloaded from balance sheet entries on Datastream.”*

From the above we deduce that the authors calculated the NCAV per share as follows:

Net Current Asset Value per share = (Current Assets – (Current Liabilities + Long Term Debt + Preferred Shares))/Common Shares Outstanding

The difference between Graham’s formula and that implied by the data used in the study being the use of “long-term debt” as a proxy for all long-term liabilities.

Lastly,* “Only those stocks with NCAV/MV higher than 1.5 are included in the NCAV/MV portfolios.”* (i.e. less than 2/3 of NCAV).

- Weighting:
*“weighted equally” and “value weighted”* - Purchase/rebalance date:
*“Portfolios of stocks are formed annually in July.”* - Holding/rebalancing period:
*“Buy-and-hold portfolios held for one, two, three, four or five years are constructed”*

While there are numerous biases/errors that can be made when conducting studies, below we have analyzed those we deem most likely to impact a study of this nature:

1. Survivorship bias

*“We include companies that have been de-listed from the exchange due to merger, liquidation or any other reason in the holding period, thus avoiding survivorship bias.”*

Based on the above the data source appears to be free from survivorship bias.

2. Look Ahead bias

*“Portfolios of stocks are formed annually in July. To be included in the sample for year t, firms must have data for NCAV in December of t-1, and at least one return observation in the post-formation period. The six-month lag between the measurement of NCAV and return data allows for the delay in publication of individual companies’ accounts, thus ensuring that the financial statements are public information before the returns are recorded.”*

Given the “six-month lag” the study appears to be free from look ahead bias.

3. Time period bias

The study spans 26 years and we classify this as a “more reliable” period.[i]

4. Data source and treatment

*“The research period is from January 1980 to December 2005 (company data prior to 1980 is unreliable and incomplete Nagel (2001)). Two databases are used: monthly return data and general information is from the London Share Price Database (LSPD), and; annual accounting data is from Datastream…Because of potential problems defining accounting variables and equity capitalisation, we exclude companies with more than one class of ordinary share and foreign companies. Also excluded are companies on the lightly regulated markets and companies belong to the financial sector…Returns for each company, including dividends, are adjusted for changes in stock splits, rights issues and stock repurchases.”*

While we do not have any specific knowledge as to the reliability of the data sets used, from the above extract it is clear the researchers took specific measures to ensure their data was reliable and that the companies, markets and sectors examined were also appropriate.

5. Human error

6. Journal rating/credibility[ii]

While academically rigorous, this study was not, to our knowledge, published in a top tier academic journal and therefore cannot be granted the “additional credibility” that may come with such publication.

**Reliability Assessment:** Based on the above the study *appears* to be reliable.

Table 2 is reproduced below:

*“We find that Graham’s NCAV/MV stocks substantially outperform the stock market over holding periods of up to five years. The average 60-month buy-and-hold raw return is 254 percent with equal weighting within the NCAV/MV portfolio and 216 percent with value weighting, which are much higher than market indices of only 137 percent and 108 percent. One million pounds invested in a series of NCAV/MV (equal weighted) portfolios starting on 1st July 1981 would have increased to £432 million by June 2005 based on the typical NCAV/MV returns over the study period. By comparison £1,000,000 invested in the entire UK main market would have increased to £34 million by end of June 2005.*

*For almost all post-formation lengths, and regardless of within portfolio weighting, the NCAV/MV portfolio outperforms either equal weighted or value weighted market indices with high statistical significance. Market-adjusted returns rise to 117 percent and 146 percent after five years if the stocks are equally weighted; and 78 percent and 108 percent after five years if the stocks are value weighted. Inspection of table 2 clearly shows that there are substantial benefits from selecting high NCAV/MV stocks.”*

Let the analysis begin!

What exactly are “Average raw returns” and are these returns truly reflective of a practitioner’s reality? “Raw returns” appear to be returns that are not adjusted for “risk” or the returns offered by the general market i.e. “market-adjusted returns”. Furthermore, the “Average raw returns” are calculated as the *arithmetic* mean[iii] of returns; consequently, the reported returns would have overstated the actual returns achieved by an investor as measured by the geometric mean (i.e. compound annual growth rate (CAGR)). We examined in detail the potential impact of the arithmetic vs geometric mean when measuring investment returns in “An Analysis of Graham's Net-Nets: Outdated or Outstanding”.

Moreover, the authors state, *“One million pounds invested in a series of NCAV/MV (equal weighted) portfolios starting on 1st July 1981 would have increased to £432 million by June 2005 based on the typical NCAV/MV returns over the study period.” *The implausibility that the strategy could have absorbed capital of such magnitude *and* achieved the reported rate of return is illuminated in our examination of the investment universe which follows.

The authors did not mandate a minimum market capitalization requirement for the securities they examined and consequently they included even the very smallest firms in the market.

They state *“…stocks are allocated to an NCAV/MV portfolio if their ratio is higher than 1.5. The numbers shown are the percentage of the average NCAV/MV portfolio falling into each size decile.… nearly 79 percent number of companies are very small (belong to size 1 and size 2)”.*

The smallest decile of the market is where 63.23% of the investment candidates were identified and it is in this decile where the greatest trading constraints are likely to be faced. From table 2 we note that “value weighting” the NCAV/MV portfolio resulted in a 4.58% (31.19% - 26.61%) reduction in the “average raw buy-and-hold” return for portfolios held for 12 months implying that the smallest stocks disproportionately contributed to the reported outperformance.

In light of the above it is highly likely that the results were biased by the inclusion of the very smallest firms as even when investing relatively modest sums, the securities of such firms are virtually untradeable.

Table 1 is reproduced below:

While the number of companies meeting the NCAV/MV > 1.5 criteria was relatively large at the commencement of the study period, by 1994 only a few companies met the necessary criteria with 1997 providing just four candidates suitable for investment. As returns were not reported for each individual year we cannot be certain of the impact on the overall returns for the years with relatively few investment candidates. This is significant given that relatively few net net investors would concentrate their portfolio in less than 10 positions. Therefore, a practitioner may wish to assume that a portion of their hypothetical portfolio was invested in cash (or cash like instruments) in years where there was a relative dearth of net nets thereby foregoing the potential returns (or avoiding a drawdown).

Furthermore, while one may be tempted to use the reported arithmetic mean returns as a guidepost to estimate the more meaningful geometric mean, we would caution against such an endeavour due to the following:

1. Psychologically, this may be the type of thinking that is driven, in part, by confirmation bias (“net nets outperform!”) and sunk cost fallacy (i.e. having put in the time and effort to read a study one may *want* to walk away “knowing something definitive”).

2. Mathematically, as demonstrated in our analysis of “Graham’s Net-Nets: Outdated or Outstanding?” the arithmetic and geometric mean can diverge materially.

3. Statistically, as the number of holdings in a portfolio falls the volatility of that portfolio may increase thereby leading to a greater potential divergence between the geometric and arithmetic mean. How portfolio volatility changes with the number of holdings in a portfolio was examined, for example, by Elton and Gruber in “Risk Reduction and Portfolio Size: An Analytical Solution”[iv] and by Alpha Architect here and here.

So, psychologically, mathematically and statistically attempting to estimate the geometric mean is precarious and more speculative than it may initially appear.

“The first principle is that you must not fool yourself and you are the easiest person to fool.”(Richard P. Feynman)

The study also sought to examine the “excess returns” of the net net strategy relative to the “UK main market” from a number of perspectives. A summary of the findings are as follows:

- Consistency – when equal weighting the NCAV/MV portfolio and value or equal weighting the market index the NCAV/MV strategy beats the market in 16 out of 20 years. Therefore, the authors conclude that
*“the strategy is fairly, but not completely, reliable.”* - Deletions and liquidations – interestingly,
*“2.6 percent of the NCAV portfolio on average failed (deleted due to liquidation) compared with 4.2 percent of the companies in the market index. This evidence does not support a risk-based explanation for the out-performance of NCAV/MV stocks, based on distress (Fama and French (1996) refer to financial distress risk).”* - Beta and standard deviation –
*“The test for CAPM-beta risk does not provide support for the view that the NCAV/MV strategy is fundamentally riskier… standard deviations of monthly returns for NCAV/MV portfolios are slightly higher than the market, but we need to consider the fact that these portfolios contain a small number of companies and so would be expected to exhibit greater volatility.”* - Size effect –
*“even after allowing for size effects in returns, there is an average NCAV/MV premium of 11.3 percent per annum for five years holding. The size effect does not fully explain the abnormal return of the NCAV strategy.”*It should be noted that while the authors state*“One million pounds invested in a series of NCAV/MV (equal weighted) portfolios starting on 1st July 1981 would have increased to £432 million by June 2005 based on the typical NCAV/MV returns over the study period”*, given*“nearly 79 percent number of companies are very small (belong to size 1 and size 2)”*(refer Table 6) it is*highly improbable*the strategy could have absorbed capital of such magnitude*and*achieved the reported rate of return. - Fama and French’s three factor model –
*“SMB, HML and the market premium do not capture the variation in NCAV stock returns.”*

Given the inability of that specified above to explain the excess returns it left the authors to suggest that *“premiums might be due to irrational pricing”*.

The study, “Testing Benjamin Graham’s Net Current Asset Value Strategy in London” appeared to be one the most reliable and rigorous studies published on net nets. Not only did the study examine the return of net nets over various holding periods, it was also the first such study to focus on the UK market, thereby creating something of an “out-of-sample” test.

Unfortunately, the “average raw returns” reported in the study were calculated as the *arithmetic* mean of returns; consequently, the reported returns would have overstated the actual returns achieved by an investor as measured by the more appropriate geometric mean (i.e. CAGR). Without specification of the geometric mean return (i.e. CAGR) we cannot be certain of the actual return achieved. In addition, the authors did not mandate a minimum market capitalization requirement for the firms they examined, consequently they included the very smallest firms in the market. The inclusion of the very smallest firms is highly likely to have biased the results as even when investing relatively modest sums, the securities of such firms are virtually untradeable.

While it is disappointing to not be able come away with a “definitive conclusion” with regard to the actual returns achieved, what we have uncovered may also be valuable; there appears to be a gap between what can be accepted in academia as an appropriate way to measure returns vs what is “truly reliable” and actually attainable from the viewpoint of the practitioner.

**Notes:**

[i] For reference: · < 10 years; inadequate/unreliable · 11 to 20 years; somewhat reliable · > 20 years; more reliable · > 40 years; most reliable [ii] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/ [iii] We were able to get in contact with the author, Glen Arnold, PhD and he confirmed that “Each post-portfolio formation month has a number ,1, 2, 3 etc. The returns are measured for the post-portfolio month e.g. month 35, for each of the portfolios starting in different years. They are then simply averaged arithmetically.” [iv] Elton, E. and Martin Gruber, 1977, Risk Reduction and Portfolio Size: An Analytical Solution, The Journal of Business 50, p 415-437.

*A version of this article has also been published by Alpha Architect **here
*

In an earlier post we analyzed the prominent and often cited study on “net nets”, “Benjamin Graham’s Net Current Asset Values: A Performance Update”, conducted by Henry R. Oppenheimer from the Financial Analysts Journal (1986). In this post we analyze the article “Graham’s Net-Nets: Outdated or Outstanding?” by James Montier. The objective of the article was to examine the performance of securities that were trading at no more than two-thirds of their “net current assets” during the 23 year period from 1985-2007 globally and regionally (namely, in the US, Europe and Japan(i)).

Our objective was to analyse the article itself; determine its reliability, draw our own conclusions and glean, if any, actionable advice for the practitioner of the “net net” method of investing.

We simply replicate the summary provided by the author for ease of reference before conducting our own analysis.

Value Investing: Tools and Techniques for Intelligent Investment, Chapter 22, p. 230/231:

- Valuation metric:
*“…current assets minus its total liabilities…”*

The author did not expressly state whether preference shares were also deducted from Current Assets; however, we assume the formula used was as specified by Graham:

i.e. Net Current Asset Value per share = (Current Assets – (Total Liabilities + Preferred Shares))/Common Shares Outstanding

*“Of course, Graham wasn’t contented with just buying firms trading on prices less than net current asset value. He required an even greater margin of safety. He would exhort buying stocks with prices of less than two-thirds of net current asset value (further increasing his margin of safety). This is the definition we operationalize below.”* i.e. the study examines the returns to stocks trading at less than 2/3 of net current asset value.

- Weighting:
*“equally weighted”* - Purchase/rebalance date:
*unspecified* - Holding/rebalancing period:
*unspecified*

While there are numerous biases/errors that can be made when conducting studies/back tests, below we have analysed those we deem most likely to impact a study of this nature:

1. Survivorship bias

Given the data source was not specified and no mention was made of controlling for survivorship bias we have no way of knowing if the sample studied may have suffered from survivorship bias.

2. Look Ahead bias

No mention was made with regard to the portfolio formation date, so we do not know if the back test suffered from Look Ahead bias, or not.

3. Time period bias

The study spans 23 years and we classify this as a “more reliable” period.

For reference:

- < 10 years; inadequate/unreliable
- 11 to 20 years; somewhat reliable
- > 20 years; more reliable
- > 40 years; most reliable

4. Human error

Human error is always a possibility; however, we have little detail on the back-testing protocol implemented so cannot comment on the increased or decreased likelihood of human error entering the testing process.

5. Journal rating/credibility(ii)

Given this was essentially an article we cannot assign it with the additional credibility afforded to publications appearing in well renowned finance journals.

**Reliability Assessment:** Given the absence of detail pertaining to the methodology and data source used to undertake the back test we deem the article to possess a **low level of reliability**. In addition, further analysis (refer below) causes us to question the reliability of the results generally.

*“An equally weighted basket of net-nets generated an average return above 35% p.a. versus a market return of 17% p.a.”*

*“Not only does a net-net strategy work at the global level, but it also works within regions (albeit to varying degrees). For instance, net-nets outperformed the market by 18%, 15%, and 6% in the USA, Japan and Europe, respectively.”*

We summarise the results in the table below (some of which were gleaned from Figure 22.2):

The author specified that the returns were “p.a.” i.e. per annum. It is not clear, to us, if “p.a.” represents the arithmetic mean or geometric mean i.e. the compound annual growth rate. This, in our opinion is of critical importance, especially given the length of the study and given it incorporated Japan, a market which experienced, to our knowledge, the greatest stock market bubble in recorded history during the period examined. But how could we find out?

Interestingly the author refers to the Oppenheimer study with which we are intimately familiar.

*“In 1986, Henry Oppenheimer published a paper in the Financial Analysts Journal examining the returns on buying stocks at or below 66% of their net current asset value during the period 1970–1983. The holding period was one year. Over its life, the portfolio contained a minimum of 18 stocks and a maximum of 89 stocks. The mean return from the strategy was 29% p.a. against a market return of 11.5% p.a.”*

Table V from the Oppenheimer study is reproduced below for ease of reference:

Using the above information, we can deduce the following with *reasonable* confidence:

The author derived “29%” by multiplying the monthly portfolio return of 2.45% by 12 i.e. 2.45% * 12 = 29.4% i.e. “29%”.

The author derived “11.5%” by multiplying the monthly market return of 0.96% by 12 i.e. 0.96% * 12 = 11.52% i.e. “11.5%”.

We know the compound annual growth rate in the Oppenheimer study for the full sample period was 28.2% (refer table IV, not reproduced here). Compounding 2.45% for 12 months yields 33.7% ((1+ 2.45%)^12-1). Therefore, we determine that the results in table V of the Oppenheimer study represent the *arithmetic* mean and not the geometric mean referred to in Table IV. Hence, based on the comparative statistics referenced by the author, we deduce, with a strong likelihood, that the author has calculated the *arithmetic* mean return.

The implication of calculating the arithmetic mean return vs the geometric mean return is stark. The arithmetic mean would, on the balance of probability, have materially *overstated* returns. Of particular concern are the returns stipulated for Japan which experienced, to our knowledge, the largest equity market bubble in recorded history in the late 1980’s which formed part of the period studied (i.e. 1985 to 2007). To illustrate the misleading nature of an arithmetic mean we refer to the “Summary Edition Credit Suisse Global Investment Returns Yearbook 2019” Table 1:

There is a material difference between the arithmetic and geometric mean achieved for all equity markets examined. For Japan the arithmetic mean is more than **double** the geometric mean! While the returns quoted are “real” returns and the time period different, our point remains, an arithmetic mean has, and does, overstate the actual returns achieved.

Moreover, a cursory glance at Figure 22.2 illustrates returns of 24% p.a. to the “USA Market” from 1985-2007; this also indicates an arithmetic mean has been used because, to our knowledge, the US market simply *did not* achieve a compound annual growth rate of 24% from 1985 to 2007!

To take our point to a theoretical extreme let’s examine a short series of returns calculating both the arithmetic and geometric mean:

Remarkably, it is *theoretically* possible to achieve an arithmetic mean twice that of the market (16.00% vs 8.00%; 2x); and simultaneously attain only a fraction of that return based on the geometric mean (4.99% vs 0.95%; 0.19x).

Given the relative dearth of studies examining the returns of net nets, especially those in Japan, we are grateful for this article. Nonetheless, due to the silence on the data source, details pertaining to the research methodology and the strong likelihood that the returns specified were calculated using the arithmetic mean, thus *overstating* returns, little can be relied on from this article.

While it may be disheartening to be left with something of a “null hypothesis”, it does provide us with the opportunity to be clichéd and conclude with a quote from none other than Charles T. Munger:

“Any year that passes in which you don’t destroy one of your best loved ideas is a wasted year.”

**Notes:**

(i) Please note, the article did not state which exchanges were examined in the markets studied.

(ii) https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

*This article was also published by Alpha Architect **here*

“Ben Graham's Net Current Asset Values: A Performance Update” by Henry R. Oppenheimer was published in the Financial Analysts Journal, Vol. 42, No. 6 (Nov. - Dec., 1986). The study examined the performance of securities that were trading at no more than two-thirds of their “net current asset value (NAV)” during the 13 year period from 1970-82 period in the US.

Please note, the terms “NAV” and “net nets” are used interchangeably throughout the analysis.

- Valuation metric

*“…sum of all liabilities and preferred stock and subtracted it from current assets; this result was then divided by the number of common shares outstanding to give NAV.”*

i.e. Net Current Asset Value per share = (Current Assets – (Total Liabilities + Preferred Shares))/Common Shares Outstanding

*“Our investor bought a security if its November closing price was no more than two-thirds of its NAV”*

- Weighting:
*“equally weighted”* - Purchase/rebalance date:
*“last business day of December of each year”* - Holding/rebalancing period:
*“The securities were held for either 12 or 30 months, depending on the analysis.”*

1. Survivorship bias

*“All securities selected from the December 1970 through December 1972 and December 1978 through December 1982 Security Owner's Stock Guide were evaluated. For the remaining years, we evaluated all NYSE securities as well as random samples of about 20 to 30 AMEX and OTC securities.5”*

Given the use of “random” samples and the Security Owner’s Guide (as opposed to a large computerized data base that might not include delisted stocks) the data source appears to be free from “survivorship bias”.

2. Look Ahead bias

*“Our investor bought a security if its November closing price was no more than two-thirds of its NAV… the date of purchase, the last business day of December of each year.”*

Given the lag between when the “NAV” was calculated and the “date of purchase” this study does not appear to suffer from “look ahead” bias.

Furthermore, given the data was retrieved from a physical manual, logically, “look ahead” bias could not have occurred.

3. Time period bias

The study spans 13 years and we classify this as a “somewhat reliable” period.

For reference:

· < 10 years; inadequate/unreliable

· 11 to 20 years; somewhat reliable

· > 20 years; more reliable

· > 40 years; most reliable

4. Human error

Given this study was created in a rather manual and laborious manner we think it prudent to assume some errors may have occurred.

5. Journal rating/credibility[i]

Appearing in the Financial Analysts Journal adds credibility to the study in light of its reputation.

**Reliability Assessment:** Based on the above and considering the clarity with which the methodology was explained the study *appears* to be reliable. However, following further analysis (refer below), we conclude that a number of findings in the study may not be as reliable as they initially appear.

Table IV from the study is reproduced below:

Below we have adapted the data from Table IV and looked at the performance in both absolute and relative terms to the S&P 500 Total Return (TR, i.e. *including* dividends):

**Gross returns**:

- Compound Annual Growth Rate (CAGR) of 28.5% (with rounding impacting our calculations) vs 9.3% for the S&P 500 TR; an absolute outperformance of 19.2% and a multiple of 3.1 relative to the S&P 500 TR
- 1 losing year (1973, which incidentally also possessed the equal smallest sample size during the study period (18 securities; refer Table I, p 41.)
- 2 negative years vs 4 negative years for the S&P 500 TR

However, as Yogi Berra is claimed to have said, *"In theory there is no difference between theory and practice. In practice there is."*[ii]

Consequently, we simulated reality by adding commissions and taxes. We included commissions of 5% (2.5% to buy 2.5% to sell)[iii]. In relation to taxes, we utilised the capital gains taxes rates of the corresponding years[iv] in the study. We assumed our hypothetical investor resided in the US and sat in the highest marginal tax bracket but held on long enough to be eligible for the long-term capital gains tax rates. In contrast we assumed an investor in the S&P 500 bought and held, never realizing their gains (and for simplicity we did not adjust their dividends for tax).

**Net returns** (i.e. including commissions and taxes):

- CAGR of 18.9% vs 9.3% for the S&P 500 TR; an absolute outperformance of 9.6% and a multiple of approximately 2 relative to the S&P 500 TR
- 4 losing years
- 2 negative years vs 4 negative years for the S&P 500 TR

In addition to commissions and taxes we also examined the impact of inflation[v], which was rampant during the period to ascertain its impact on returns.

**Net Real returns** (i.e. net of commissions, taxes and adjusted for inflation):

- CAGR of 10.7% vs 1.4% for the S&P 500 TR; an absolute outperformance of 9.3% and a multiple of 7.4 relative to the S&P 500 TR
- 4 losing years
- 2 negative years vs 5 negative years for the S&P 500 TR

Table VI from the study is reproduced below:

*Note: It is unstated whether this “mean monthly return” figure is intended to be a geometric or arithmetic mean.* *Oppenheimer “put[s] [the] results in a form more meaningful to an investor” by stating that “$10,000 invested in the NAV portfolio on December 31, 1970 would have grown to $254,973 (with monthly compounding) by December 31, 1983.”* *(p. 42.)* *However, that is a period of 156 months, and compounding $10,000 at 2.45% for 156 periods leads to a terminal value of $436,381, not $254,793.* *The implied compound monthly return rate for 156 periods and Oppenheimer’s terminal value is 2.10%.* *Consistent with this 2.10% figure, Oppenheimer states explicitly that the geometric mean annual return for the 13-year period was 28.2%.* *(p. 44.) Therefore, we conclude that the table above displays the arithmetic mean. *

With regard to Table VI the author states; *“The conclusion is clear-cut. Returns and excess returns can be rank-ordered, with securities having the smallest purchase price as a percentage of net asset value having the largest returns. It appears that degree of undervaluation is important…”*

Notwithstanding the above, due to concerns over the data used to measure returns (which is discussed in detail in subsequent sections), based on this study alone (and *not* taking into account our wider body of knowledge), the conclusion “that the degree of undervaluation is important” is *not* as reliable as it initially appears (despite its likely accuracy).

Table V from the study is reproduced below:

When it comes to dividends the spread between “Positive Earnings and Dividends” (Panel D) and Positive Earnings and No Dividends (Panel E) is stark. No economic rationale comes to mind for why a net net paying dividends would underperform one that does not pay dividends. Perhaps the spread is due to positive earning dividend payers possessing a shallower discount compared to positive earnings non-dividend payers. For instance, if positive earning dividend payers were consistently in the “most expensive” quintile (Table VI, Panel A) then their returns of 2.01% are similar to the 1.88% identified for the most expensive quintile. Similarly, if positive earning non-dividend payers were consistently in the “cheapest” quintile (Table VI, Panel A) then their returns of 2.88% are not materially different from the 2.95% identified for the cheapest quintile. That said, the aforementioned seems unlikely.

Interestingly in note 17 (p. 47) of the study the author specifies that *“An analysis of the 13 portfolios was also performed. The conclusions about earnings and dividends also hold for them”* implying an element of robustness to the conclusion that *“Firms having positive earnings and paying a dividend provided a lower mean return than portfolios of firms with positive earnings not paying a dividend. They also had a lower systematic risk. Finally, their risk-adjusted excess returns were not as high as those of the portfolio of firms with positive earnings but not paying dividends. Choosing only firms that have earnings and pay a dividend does not help the investor.”*

It would appear then, that dividends, at least in relation to the period and market studied were a hindrance to returns…or were they?

Note 9 (p. 47) specifies the following:

*“all comparisons with the exchange benchmarks returns without dividends are used for both the security return and the benchmark return.”*

That is to say, to our understanding, the dividends have been excluded in the returns displayed in Table V (among others), and therefore the returns have been artificially reduced by the amount of the dividend paid out. On this basis it is not at all unexpected that the dividend payers would seemingly underperform their non-dividend paying counterparts.

For clarity we will demonstrate using an example between a non-dividend and dividend paying stock:

From the table above Stock B (the “dividend payer”) has produced the higher “total return” (i.e. 50% vs 30%), however, if you exclude the dividend Stock A (the “non-dividend payer”) *appears* to have produced the larger return (i.e. 30% vs 20%). While perplexing, it does, to our comprehension, appear as though this rudimentary error (i.e. excluding dividends) has been made in conducting this study[vi].

In light of the above, we are of the opinion that no reliable conclusion can be drawn in relation to the impact of dividends on the return to “net nets” based on this study.

The author sees the differential between “Positive Earnings” (Panel B) and “Negative Earnings” (Panel C) as marginal. The author concludes that *“No clear-cut pattern emerges from an examination of these panels. If anything, firms operating at a loss seem to have slightly higher returns and risk than firms with positive earnings.”*

However, in light of our findings with regard to the data used to measure returns (discussed in depth in the section titled “Dividends”) we deviate from the author’s assessment in relation to “positive” and “negative” earning net nets. We agree that “*No clear-cut pattern emerges from an examination”;* but do so for different reasons.

It is possible (probable in our view) that “positive earners” had a greater propensity to pay dividends. Therefore, if those dividends were *not* included in the measurement of their returns they would be artificially reduced. Of course, it is also possible (albeit unlikely in our view) that “negative earners” paid the greater share of dividends subsequently understating their returns. Irrespective, we are of the opinion that no reliable conclusion can be drawn in relation to the impact of earnings on the return of “net nets” based on this study.

Table III is reproduced below:

While it would have been interesting to analyze the returns for portfolios held for 12 and 30 months this is not possible as the annual returns to the 12 month holding/rebalance period per Table IV are geometric means, whereas the returns displayed in Table III above are arithmetic means. However, the author’s commentary does specify that *“The results largely parallel those of Table II” *[which displays the monthly arithmetic mean for a 12 month holding/rebalance period].* From a return (and wealth) perspective, the advantage of the NAV portfolios over the market indexes is consistently pronounced.”*

January effect: the author tested and found the return to the NAV portfolios *“while consistently smaller than those reported in Table III, still supported the conclusion that the NAV criterion provided satisfactory performance.” *That is to say, the performance of the NAV portfolios was not solely explained by the “January effect”.

Size effect; to demonstrate that the returns to the NAV strategy were not merely a result of the “size effect” the author provides analyses (p. 45.) examining the return to the small-firm index and the “MV1” portfolio which is even smaller than the NAV portfolios. While we do not know the valuation of the small-firm index and the MV1 portfolio the conclusion that the returns to the NAV portfolios are not purely a result of the size effect seems reasonable.

However, no minimum market capitalization cut off for the securities examined in this study was mandated. Furthermore, we note that the reported median market capitalization of securities in the portfolios examined was 4.1m. That is to say, half the firms examined had a market capitalization equal to or below 4.1m which is particularly small, even by today’s inflation adjusted standards. The securities of the very smallest firms within a market are, in practice, virtually untradeable, even when attempting to deploy relatively modest amounts of capital.

In light of the foregoing it is highly probable that a considerable number of firms included in the study were, in effect, uninvestable. Consequently, the reported returns are highly likely to have been materially biased upwards given the findings reported in other research pertaining to the very smallest stocks within a market and their associated lack of liquidity[vii].

One noteworthy comment that stands out from the study is as follows: *“Even though the NAV criterion is the valuation technique Graham is most famous for, it has been subject to relatively little research.' Oppenheimer has provided tests of its performance over the 1949-72 period, but his tests (which do not demonstrate consistent profits) are largely confined to data prior to 1958.” (p. 40)*

Interesting; unfortunately to date we have not found these other “tests”.

“Ben Graham's Net Current Asset Values: A Performance Update” examined the performance of securities that were trading at no more than two-thirds of their “net current asset value (NAV)” during the 13 year period from 1970-82 period in the US. The study reported gross returns that were more than triple the returns of the S&P 500 TR. Our simulation determined that the aforementioned returns, on a net basis (i.e. after commissions and potential taxes) would have nearly doubled those generated by the S&P500 TR.

However, the study utilised data that excluded dividends (i.e. “price return” data as opposed to “total return” data) and therefore the conclusion reached in the study, *“[c]hoosing only firms that have…. a dividend does not help the investor”* does not appear to be reliable. Furthermore, as data excluding dividends was also used to measure the returns of “positive” and “negative” earning net nets, no reliable conclusion can be reached in relation to the impact of earnings on the return of net nets either[viii]. Critically, it is highly probable that the study included a considerable number of firms whose securities may have been too illiquid to actually trade. Moreover, the inclusion of such firms is highly likely to have biased the reported returns upwards.

While the study initially appeared to be reliable, in reality it suffered from a number of material flaws such that it is highly improbable that the reported returns could have been achieved by an investor in practice.

**Notes:**

[i] https://alphaarchitect.com/2015/04/01/where-to-find-cool-academic-finance-research/

[ii] The quote, *“in theory there is no difference between theory and practice, while in practice there is” *is more likely to have been stated by Benjamin Brewster.

[iii] A Century Of Stock Market Liquidity And Trading Costs, Charles M. Jones, Graduate School of Business Columbia University, 2002. (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=313681)

[iv] https://taxfoundation.org/federal-capital-gains-tax-collections-1954-2009/

We assumed our investor was in the highest tax bracket.

For 1978 long term capital gains tax rates were 39.875/33.85%, we adopted the lower of these rates.

For 1981 long term capital gains tax rates were 28/20%, we adopted the lower of these rates.

[vi] While we attempted to contact Mr. Oppenheimer to attain clarification on the matter, unfortunately, he appears to have retired in 2016. (https://web.uri.edu/business/2016/02/10/professor-oppenheimer/)

[vii] James O'Shaughnessy, What Works on Wall Street: The Classic Guide to the Best-Performing Investment Strategies of All Time, 4th ed. Chapter 5 (New York: McGraw-Hill, 2011).

[viii] In and of itself the exclusion of dividends would have understated the reported returns.

*The original version of this article was also published by Alpha Architect **here.*