By on January 4, 2013

If your life, career, wealth, or all of the above depend on forecasting the auto market with precision and reliability, then you can do no better than securing the services of Edmunds’ chief analyst Jessica Caldwell. Last year’s winner of the coveted  “TTAC  11” (a.k.a. “The Top Analyst Crown 2011”)  does it again: Jessica Caldwell wins the TTAC 12, closely followed by her perennial antagonists, Jesse Toprak of TrueCar and Peter Nesvold of Jefferies.

TTAC congratulates  Jessica Caldwell and her team to her second coronation in a row.

TTAC Analyst Ranking 2012, Final Standings
Rank Analyst Average Deviation
1 Jessica Caldwell (Edmunds.com) 12.12%
2 Jesse Toprak (TrueCar.com) 15.55%
3 Peter Nesvold (Jefferies) 17.08%
4 Alec Gutierrez (Kelley) 18.08%
5 Brian Johnson (Barclays) 24.60%
6 Patrick Archambault (Goldman) 35.40%
7 Joseph Spak (RBC) 40.45%
8 Rod Lache (Deutsche Bank) 48.10%
9 Chris Ceraso (Credit Suisse) 49.43%
10 Emmanuel Rosner (CLSA) 80.80%
11 John Sousanis (Ward’s) 110.95%
12 Alan Baum (Baum & Associates) 302.64%
13 Jeff Schuster (LMC Automotive) 302.85%
14 George Magliano (IHS) 311.23%
15 Adam Jonas (Morgan Stanley) 342.89%
16 Itay Michaeli (Citigroup) 359.90%
17 Matthew Stover (Guggenheim) 383.83%
18 Himanshu Patel (JPMorgan) 384.23%
19 Christopher Hopson (IHS) 391.83%

Each month, Bloomberg asks a panel of America’s best and (hopefully) highest paid analysts how this month’s auto sales will come in. Each month, TTAC grades the analysts on how well they hit their target. Each month, some of our readers ask why.

The auto business lives on reliable forecasts. Marketers and engineers have to gauge customer desires many years ahead. Budgets have to be made, resources allocated, production schedules have to be put together, parts must be ordered, advertising dollars allocated, cars shipped to dealer lots. What’s more, car sales are an indicator for the economy.  That’s why brokerage houses and banks employ hordes of analysts, that’s why specialist firms like Edmunds, TrueCar, Kelley, IHS and LMC invest considerable energy into forecasting the market. And that’s why TTAC benchmarks analyst results.
Nobody can predict the future with 100% accuracy. Panels of reporting dealers can be built that bring you pretty close to the actual results. Fleet sales make the matter tricky. A volume deal, moved a day back or ahead can throw off the best forecast.
When you look for a good forecaster, you look for reliability and consistency. The hero is not the forecaster who hits a bulls-eye once.  A good forecaster is close to the truth most of the time. Jessica Caldwell is a great forecaster. A good forecaster never takes a vacation, he or she never shrugs the shoulders and says “I don’t know.”

If I would have to recommend the perfect forecaster, then I would name a panel consisting of Jessica Caldwell, Jesse Toprak of Truecar, and Peter Nesvold of Jefferies.  Between them, they have six first places last year, eight second places and  five third places.  Actually, I would have a chat with Brian Johnson of Barclays and put him on the panel if he promises never to punt again. In April last year, Brian shrugged his shoulders when it came to Chrysler and said “I don’t know.”  This slip-up did cost Brian a podium finish in the annual TTAC ranking. Did we mention consistency and reliability?

TTAC Analyst Rankings December 2012
Rank Analyst  GM Ford  Chrysler  SAAR SAAR Diff OEM Diff Overall
1 Brian Johnson (Barclays) 6.7% 0.6% 10.0% 15.5 0.8% 3.1% 3.9%
2 John Sousanis (Ward’s) 4.0% 1.0% 9.0% 15.7 2.1% 3.0% 5.1%
3 Jessica Caldwell (Edmunds.com) 1.5% 1.6% 7.3% 15.4 0.2% 6.6% 6.8%
4 Chris Ceraso (Credit Suisse) 2.0% 3.1% 8.1% 15.5 0.8% 6.0% 6.8%
5 Jesse Toprak (TrueCar.com) 0.9% 2.6% 8.8% 15.6 1.5% 5.9% 7.4%
6 Joseph Spak (RBC) 2.6% 0.8% 7.1% 15.6 1.5% 6.5% 8.0%
7 Rod Lache (Deutsche Bank) 3.0% -2.0% 9.0% 15.1 1.8% 7.0% 8.8%
8 Peter Nesvold (Jefferies) 0.0% 1.2% 7.3% 15.3 0.5% 8.5% 9.0%
9 Patrick Archambault (Goldman) 1.4% 3.0% 6.4% 15.5 0.8% 8.2% 9.0%
10 Emmanuel Rosner (CLSA) 0.3% 0.7% 5.1% 15.4 0.2% 10.9% 11.1%
11 Alec Gutierrez (Kelley) 0.4% 0.3% 5.9% 15.2 1.1% 10.4% 11.5%
12 Ryan Brinkman (JPMorgan) NA NA NA 15.4 0.2% 300.0% 300.2%
13 Alan Baum (Baum & Associates) NA NA NA 15.4 0.2% 300.0% 300.2%
14 Jeff Schuster (LMC Automotive) NA NA NA 15.3 0.5% 300.0% 300.5%
15 George Magliano (IHS) NA NA NA 15.2 1.1% 300.0% 301.1%
Average 2.1% 1.2% 7.6% 15.40
Actual 5.0% 2.0% 10.0% 15.37

Case in point: In December, Brian Johnson took the top post, with Caldwell in place three, Toprak in 5 and Nesvold in eight. Brian took two first last year, one second and two thirds. Would he not have chickened out on Chrysler last April, Johnson probably would have a podium finish in the annual rankings in addition to his first place in December.

How does one predict sales with accuracy, reliability and consistency?  Apart from the obvious (guess, call the OEMs, establish a real time dealer panel, knock on wood) the intricate details remain a closely guarded secret.  Asked how in the world she can be so good, Jessica Caldwell delivered a politically correct statement that nonetheless won’t help you break into the forecasting business:

“Forecasting is, of course, a team effort. Out team includes several PhDs, statisticians and hundreds of other colleagues who are working to translate this data and deliver key insights to help both shoppers and dealers improve the car buying experience. You won’t find a more hard working group of data experts anywhere else in the industry.”

A word to the also-rans, beginning halfway down the annual list: Only betting on a SAAR number and withholding an estimate for the Detroit Three is a surefire way of ruining your chances of winning.  This has sunk in. In January,  six out of 14 analysts only provided a SAAR.  In December, this number was down to four out of 15. Likewise, analysts who pop up only once or twice in the year stand no chance. It would be better to bet in the name of the vacationing colleague. This is a highly competitive and unforgiving business.

A spreadsheet with the monthly results and other nifty stats can be downloaded here.

Get the latest TTAC e-Newsletter!

14 Comments on “Grade The Analysts: Jessica Caldwell Crowned Most Reliable Analyst Of 2012...”


  • avatar
    grzydj

    In all of her infinite wisdom, she wasn’t able to predict the demise of Edmunds own insideline.com?

    It’s a shame too, because it was my second favorite site, right after TTAC of course.

  • avatar
    Landcrusher

    There are some sad stories after 9th place.

    I would be really interested to hear more about the big bank analysts near the bottom. If they are that bad, why aren’t their jobs outsourced?.

  • avatar
    jimboy

    Excellent work Jessica! Well done.

  • avatar
    Pch101

    The 300% thing is completely bogus. You really ought to remove that, as it makes no sense — you’ve arbitrarily decided that someone who doesn’t bother to predict Detroit’s sales each month is “bad” with a 300% error rate.

    As for the SAAR, if I take the second table at face value, then I have to conclude everybody’s calls were fairly accurate. All of them were within 2% of the actual figure. There is no meaningful difference between any of these.

    • 0 avatar

      Someone who does not make a forecast is 100% off. Omit three forecasts, you are 300% off. We have been doing this like that for more than a year and had the algorithm reviewed by major forecasters.

      It will stay that way.

      Even I could guess the SAAR with reasonable precision. I would not dare predicting the D3 …

      • 0 avatar
        Pch101

        “Someone who does not make a forecast is 100% off.”

        No. Someone who didn’t make a forecast didn’t make a forecast. This 300% thing is statistical abuse at its finest.

      • 0 avatar
        drivelikejehu

        With respect, I do not see how it can be justified statistically. It makes more sense to either omit those that make incomplete predictions, or assume they view all automakers as identical and project the D3 with the SAAR differential.

        The latter approach makes sense math-wise but, for shaming purposes, they should probably just have their own list for incomplete predictors.

      • 0 avatar
        Landcrusher

        I gotta agree with PCH on this one. I thought you were adding the monthly percentage misses. It’s very misleading.

      • 0 avatar
        Georgewilliamherbert

        Yeah, this seems statistically bogus for general news reporting. It seems highly misleading. Do they ever report those values, or once a quarter, or intermittently for some months, or what? If you score them on a different time range than they actually report on, then your reports are somewhere between (unfair and confusing) and (bogus).

  • avatar
    prndlol

    Alright Jessica, what car am I thinking about…right now!

  • avatar
    AoLetsGo

    First Congrats to Jessica and her team. Myself and others at my company do some sales forecasting for clients and we consider anything within +/- 15% a very good score. If I say so myself our job is tougher since we get one shot at a sales number (no monthly revisions) and it can be a year or two out in the future. That said we tell people that while we strive to be 100% accurate what is really important is that we are directionally correct and provide good guidance on go or no go decisions.

  • avatar
    GiddyHitch

    Jessica and her team obviously do great work but I am now wondering how much of her forecasts are driven directly by data gathered from the Edmunds website itself by consumers shopping for a new vehicle. This would be an inside line on potential sales unavailable to other analysts.


Back to TopLeave a Reply

You must be logged in to post a comment.

Subscribe without commenting

Recent Comments

New Car Research

Get a Free Dealer Quote

Staff

  • Authors

  • Brendan McAleer, Canada
  • Marcelo De Vasconcellos, Brazil
  • Matthias Gasnier, Australia
  • W. Christian 'Mental' Ward, Abu Dhabi
  • Mark Stevenson, Canada
  • Faisal Ali Khan, India