How to build a startup evaluation toolkit

In the first installment of our startup evaluation guide, we cover five essential tools you need to ensure you are heading in the right direction.

tools or toolkit surrounding a lightbulb on a chalkboard to build, develop or repair
phaisarn2517 / Getty Images

Over the years, I’ve built a toolkit I use to evaluate startups anytime I get lost amidst all of the jargon and hype. In this first startup evaluation guide, I’m going to cover the essentials, the five key tools that I use the most often.

The first tool is the most essential one, the one I turn to over  again to make sure I’m heading in the right direction.

Tool No. 1: The compass, to keep you oriented to true north

Gathering useful data points on startups is difficult. Some stay in stealth-mode for years. Others promote themselves far and wide, but with hype, noise and exaggerated claims. These startups only hint at why their technology is worth considering, holding their cards much too closely to their vests, and evading your questions when you try to dig too deep.

Even when a startup if fairly transparent, they are still difficult to assess because they lack the common data points we typically use to evaluate products in a crowded market: Most importantly, they lack testimonials and other types of third-party validation.

Most startups do not yet have on-the-record customers; many obfuscate what exactly

During the first years of this new millennia, I was a newcomer to the high-tech scene. I had just landed a job editing insider tech investment newsletters that advised venture capitalists (VCs), hedge funds and other institutional investors on the best startup bets in an overheated market.

The learning curve was steep, and just as I felt I was getting reasonably good at separating the wheat from the chaff in the fund-anything-that-moves dotcom market, the bubble burst.

Related: CIO Quick Takes: How to evaluate startups

In April 2000, the NASDAQ lost 20 perecent of its value in a single week. Most of us earning a living off of the dotcom fever – even us newbies – already understood that the party had to end eventually. Still, the abrupt drop was a shock to all but the most ardent skeptics.

In the nearly two decades since the dotcom bubble burst, I’ve learned a lot about startups and the environments they compete in. I’ve spent a good chunk of those years working as a tech journalist, writing for such publications as Network World, CIO, Cloudbook, Datamation, Forbes, Wired, Telecom Trends, and others.

I’ve also written for, worked in, and consulted for technology companies both large and small, including Aryaka Networks, Big Panda, Citrix, Esri, Microsoft, Qualcomm and Xerox, to name only a few. And I’ve been lucky enough to work on a few very cool projects that I can’t talk about (NDAs), but wish I could.

There are also an equal number of downright bonkers projects I would like to roll out as cautionary tales, but those are a good reminder that NDAs aren’t always about protecting sensitive IP. Reputations are every bit as important to protect if your startup hopes to avoid being roadkill in a hypercompetitive market, but protecting reputations can go too far, pulling startups off course.

Related: Where startups are impacting long-term change for CIOs

Over the years, I’ve built a toolkit I use to evaluate startups anytime I get lost amidst all of the jargon and hype. In this first startup evaluation guide, I’m going to cover the essentials, the five key tools that I use the most often.

The first tool is the most essential one, the one I turn to over  again to make sure I’m heading in the right direction.

Tool No. 1: The compass, to keep you oriented to true north

Gathering useful data points on startups is difficult. Some stay in stealth-mode for years. Others promote themselves far and wide, but with hype, noise and exaggerated claims. These startups only hint at why their technology is worth considering, holding their cards much too closely to their vests, and evading your questions when you try to dig too deep.

Even when a startup if fairly transparent, they are still difficult to assess because they lack the common data points we typically use to evaluate products in a crowded market: Most importantly, they lack testimonials and other types of third-party validation.  

Most startups do not yet have on-the-record customers; many obfuscate what exactly they do, and all of them are fighting an uphill battle, since the majority of startups will fail within five years.

What you’re left with is an incomplete picture, and most technology buyers sensibly rule out opaque startups well before the short list. The lack of transparency is something journalists, analysts and, in many cases, even investors must cope with, as well.

Over the years, I’ve sought tools to include in my toolkit that help me crack open the black box that is a startup’s true fundamentals. There are many different ways to evaluate startups. Talk to 10 different VCs and you’ll get 10 different answers about the most important data points to seek.

What most gloss right over, however, is the importance of orienting yourself within a sea of noise. Startup founders are experts at pulling you away from your needs to focus on theirs. The best startups have compelling origin stories, colorful founders, and intoxicating visions of a tech-driven future.

If you don’t want to get suckered into vapor- and shelf-ware, however, you need a compass. You need to constantly orient yourself back to your and your organization’s needs and goals. Find your true north and stick to it, reorient yourself often, especially when on unfamiliar terrain, and when the noise is too much too tune out, pull out the second essential tool from your eval kit.

Related: 10 success techniques of high-growth startups

Tool No. 2: Earplugs, so you can ignore the hype

Gathering useful data points on startups is difficult. Most don’t have on-the-record customers; many obfuscate what exactly they do, and all of them are fighting an uphill battle, since, again, the majority of startups will fail within five years.

Until a high-tech product is out in the market where it is used by real people in a production environment where it runs up against real-world conditions, the viability of the startup’s product or service is next to impossible to evaluate.

Filling the gap is a torrent of noise, hype and vendor spin.

I could advise you to just “ignore the hype,” but that’s much easier said than done, especially when hype is the only available information. Thus, a well-fitting, comfortable set of ear plugs is essential. (I’m speaking metaphorically, of course, but just barely.)

Much of the propaganda has devolved to become a table-stakes nuisance. Pre-product startups refer to themselves as “leading providers,” and no one bats an eye because disinformation and spin flood the information zone. Plenty of established vendors do this as well, spewing out their own overblown claims and launching FUD (fear, uncertainty and doubt) campaigns against any competitors nipping at their heels, but at least the incumbents have reference customers you can talk to.

Those earplugs aren’t always going to be enough, however.

Not only is vendor-hype often the only information startups give you to work with, but hype can also be so infectious that it spreads like kudzu over entire market sectors. For hype-fueled startups, truth is replaced by “truthiness,” and in overheated markets, incumbents often follow startups’ leads, believing they must keep up with the hype or risk becoming obsolete, slapped with the dreaded “legacy” label. As a result, enterprise tech buyers are often forced to make their choices based on deeply flawed information.

The dotcom craze is a good example of this. Cloud, the gig economy, social networking, unified communications, and the internet of things (IoT) are all sectors where hype, BS and rampant exaggeration have flooded the messaging zone at one time or another.

Thus, it’s important to complement your ear plugs with essential tool No. 3.

Related: How the earliest of tech startups are surviving

Tool No. 3: A finely tuned BS detector

“News is something somebody doesn’t want printed; all else is advertising,” William Randolph Hearst, himself a legendary BS merchant, once said. Tech journalism is slightly different, since much of what we’re trying to do is educational, rather than investigative. Personally, I’ve spent far more time introducing readers to trends, rather than digging to expose dark industry secrets. But Hearst’s guidance still applies, and I’ve always felt that my first job when introducing readers to a new startup is to figure out what the startup doesn’t want you to know.

A finely calibrated BS detector is a journalist’s best friend. Heck, you need a good BS detector to avoid getting ripped off in any market, but it’s doubly true in tech. Founders tend to be specialized experts in their fields, whereas tech buyers (and journalists, for that matter) tend to be generalists.

Our natural response is to defer to the specialists when we’re wading into new territory, but what we neglect at our peril is the fact that specialists not only often succumb to tunnel vision, but they also have their own agendas, agendas don’t always align with the truth.

There are easy ways to fine-tune your B.S. detector. Search for buzzwords and determine whether their use adds precision or empty calories to a startup’s messaging. Ask follow-up questions whenever the startup dodges your need-to-know questions. Seek out vagueness, jargon and buzzwords, and probe those areas doggedly.

Often, vagueness, jargon and buzzwords are used to paper over weaknesses. If a startup won’t go deeper than the buzz, rule them out and move on.   

As startups raise more funding, attract beta customers and add seasoned business leaders to their leadership team, they get harder and harder to pin down. A really good CEO, after all, is an expert at messaging. The hype will start to be balanced by decent information, but that “transparent” seeming information often is anything but. Rather, it is industry table stakes trotted out as “special sauce,” or the generally available information on, say, how software-defined networking functions, is just a spruced up, slightly revised Wikipedia entry.

Basic industry information is used to distract you from the lack of innovation layered on top of open-source, or commodity, technology. In these cases, ear plugs and a BS detector may need to be augmented by your fourth essential tool. 

Tool No. 4: The machete. . . or a red pen will do in a pinch

As a journalist, my first point of contact is nearly always with PR representatives, whose very existence is built on spin, hype and amplification. Worse, that first point of contact is also usually unsolicited and tends to have a look-at-me, look-at-me, tone-deaf quality.

Most pop into your inbox with a big ask (give us coverage!) before they’ve even bothered to learn my name. I can’t tell you how many pitches I get that start with a generic “Hello,” the “To whom it may concern” of our informal era. Many of these messages have automated follow-ups, as well, which typically open with such nuisances as, “Hello, I just want to check back and see when you’d like to interview our founder;” “Hey there, I want to circle back to make sure you saw this industry-changing news;” or the worst one of all: “Hello, Just want to bring this important news back to the top of your inbox.”

If a PR person has the ability to automate shotgun-blast pitches, but can’t personalize those pitches, it’s a red flag, and my response is always the same: delete. My machete, thus, isn’t a real machete, but the editor’s equivalent: the delete key and the red pen.

Now, you’re probably thinking, “Hey, I’m not a journalist.” That’s true; your getting-to-know-you pipeline will have different information, say, starting with a white paper, followed by an educational email nurturing campaign, before, perhaps, pulling you in to read a product data sheet or website copy.Your red-pen machete will work equally well on a white paper as a press release.

In my non-tech life, I used to teach undergraduate and graduate-level writing courses. A trick I use to teach students how to make not only their writing, but also thinking, sharper is to print out their essay, article, or story. Next, take a red pen and cross out every adverb. Adverbs tend to be band-aids on weak verbs.

Then, eliminate every adjective. These tend to precede weak nouns. Then, we hack out prepositions and other modifiers that aren’t supporting their own weight.

Finally, go through what’s left – mostly nouns and verbs, along with a few determiners and conjunctions – and replace the words tasked with doing the real work with more precise, stronger ones. Of course, this is a lesson and you can only extend it so far. I use adjectives and adverbs all the time in my own work, but I do try to be selective. 

This trick works equally well for freshmen intro to composition courses as with graduate-level creative nonfiction ones. It’s a trick I use myself from time to time, whenever I feel that my writing is getting too loose and flabby.

It’s also a technique I’ve used right in front of startup CEOs on a handful of rare occasions when I got so frustrated by the relentless spin, evasion and disinformation that I hacked up their collateral during the briefing to see if anything concrete was left.

That’s more or less the nuclear option, so I wouldn’t recommend it for any but the most severe cases of vendor-spinitis.

As a mental shortcut, though, this approach is invaluable when evaluating the “educational” information startups throw your way. I don’t pull out a red pen, but I’ve trained myself to eliminate all of the noise words when I read PR pitches, press releases, case studies, white papers and other copy.

Try it sometime. You’ll be surprised by how little is left after you’ve hacked away kudzu phrases like “leading provider,” “revolutionary” and “cutting edge” from their copy, revealing its bare essence.

Tool No. 5: The oxygen sensor

I could fill volumes with PR horror stories, but truth be told, I actually consider a lack of professional PR and marketing to be a bigger red flag than bad PR.

Startups that ignore PR and marketing risk permanent obscurity.

My fifth essential tool in my startup evaluation kit is the oxygen sensor. As a mechanic’s son, I learned the value of oxygen sensors at an early age. A faulty oxygen sensor, although only a small, inexpensive part, can do big-time damage, killing your gas mileage, causing a rough idle, and even degrading into engine misses.

Say what you will about the PR industry, and I’ve said plenty myself over the years, but I must concede that PR pros know how to relentlessly pitch stories, pester journalists with endless followups, and when they earn coverage, they know how to amplify it.

If you want to assess a startup’s potential longevity, evaluating a startup’s PR and marketing are an unavoidable first step. I can’t name a single startup that knocked off an incumbent without a competent PR and marketing effort.

I can think of plenty of startups, on the other hand, with inferior, but good-enough, tech who jumped to the top of the heap through expert PR and marketing campaigns. Think about it. Fires need oxygen to burn, and if a startup has an amazing story worthy of going viral, they still need an experienced marketing team to light the spark and a competent PR team to fan the flames.

Good PR can elevate a startup from mediocre to good, and from good to great. What it cannot do is turn a loser into a winner. Sure, PR can paper over weaknesses for a while, but if the startup doesn’t correct those weaknesses, the PR-glow will wear off quickly.

Conclusion: When evaluating startups, first find true north within the noise

Evaluating startups isn’t easy. Critical information is shielded behind NDAs, the “proprietary” label, evasion and disinformation. However, even evasion and disinformation are data points, if you know how to navigate through them.

The five essential tools listed above are only a start, but they will orient you, eliminate noise and ground you in the concrete. In our second guide in this series, we will move on to understanding the critical problems startups intend to solve, evaluating the startup’s solution from the customer back to the technology. . . which is the exact opposite approach from the one startups will try to convince you to take.