We at
commercetools
use dozens of software vendors, from
PagerDuty
, to
Humio
and behind. Traditional RFPs where you ask hundreds of "Do you support [granular feature X]" are pretty worthless. In any mature category, the top handful of vendors all have the same baseline of features and can pretty easily handle most use cases. Instead, here are the 25 questions I ask of potential mission-critical vendors:
- Is the product strategy largely fixed, or is the vendor still trying to find product/market fit?
- How does this vendor roadmap? What is its high-level roadmap? Good companies don’t don’t always do what their customers say (for good reason), but they at least collect feedback qualitatively and quantitatively
- When features are announced, does that mean the features are actually available to be used? Or are they announced months in advance for marketing purposes only?
- Are you a “typical” customer (vertical, GMV, ACV, etc) for this product or an outlier?
- How does the vendor communicate its new features, roadmap, change in strategy, etc?
- Have there been any major security or availability incidents? Where/how are these communicated?
- Does the vendor quantify its tech debt? How? How much debt does the vendor have? What are the implications of this tech debt to you as a customer?
- Is the product you’re buying the vendor’s sole focus? Or is this product a new foray for the company?
- What % of customers self-implement? If going 3rd party, what’s the multiple of license revenue to implement?
- Has the vendor recently converted to MACH? Go really deep and understand the depth and breadth of their conversion. The vast majority of these “We’re now MACH” vendors are lying to you. Check if the vendor has been admitted to the MACH Alliance
- Are there any hard dependencies on open source? If a key framework / project changes its licensing model, is that going to severely impact the product or your ability to legally use the product?
- Where is the vendor in its lifecycle, from seed -> being put out to pasture by a legacy incumbent? Is it growing and looking for logos? Or is it squeezing its existing install base for every last dollar?
- How much has the vendor raised to date? Is the vendor financially solvent? What does cash burn and runway look like?
- How active are the investors? Is this a founder-led vendor, whose founder has a clear vision and the ability to deliver on that? Or an investor-led vendor, with the CEO being a hired employee?
- What does the ISV and SI partner ecosystem look like? Where does the vendor compete with its ISV and SI partners?
- How quickly has the vendor grown or declined over the past 6, 12, 18 and 24 months? If there have been any layoffs, understand why
- Has the vendor ever been sued by one if of its customers or partners? If so, understand the context and judge for yourself
- How does employee turnover look? Has management turned over a lot? How do Glassdoor reviews look?
- How does the vendor handle outages? Have their CTO walk you through the process they use internally. Look for a formal policy, designated points of contact, ongoing drills, and a formal place for customers to see up-to-the-second updates
- What are the various support levels? Is first tier support outsourced? What is the evaluation criteria used to escalate tickets?
- Is there a customer success function? How many customers does each csm handle? Does the csm team handle renewals? Will you have a dedicated contact? Where are your escalation points?
- Who are your executive escalation points if things go wrong? Not necessarily the CEO, but someone who’s deeply involved at the ground-level and can work across a vendor to fix your problem. This person is typically a CTO, CPO or similar
- What does the vendor expect of you once you’re a customer? Do they want to use your logo? Will you be expected to serve as references for other customers or analysts?
- What does the licensing model look like? Are there any charges overages?
- What are internal SLOs and external SLAs? What’s included in the SLA? For example, is business user tooling included? Or background processes? If the vendor doesn’t meet the SLA, do you get money back? Will the vendor credit you automatically or do you have to track outages and request the credits?
I hope these questions can be of help out there as you're selecting new vendors.
Global Head of Solutions Architecture (Pre-Sales)
1yI love #4. MANY years ago we were looking for a radiology information system, and I remember my boss saying, “if we find anyone who offers exactly what we need off the shelf, we don’t want to buy it, because we’ll be their only customer.” As it turns out, we did find the exact fit, and basically did the old Remington commercial, “I liked it so much, I bought the company!” routine.
STRATACTICAL™ MARTECH MAXIMIZER → Digging in where strategy, tactics & tech converge @Salesforce
1yAll good questions, Kelly Goetsch! My favorite is #4: "Are you a 'typical' customer (vertical, GMV, ACV, etc) for this product or an outlier?" In my experience, sometimes being an outlier can be an advantage, especially by vertical, because you can help shape the solution to the industry. But it requires positive answers to most of the other great questions on your list to have being the guinea pig work out.
VP @ Luminos Labs 🔗 | B2B Technology Executive 🏭 | Business Partner 🤝 | Legacy to Modernization Architect 📈
1yAgreed, really appreciate focusing on non-functional aspects because that's what changes things from a "vendor" to a "partner". Checkbox commoditized features are table stakes; this means that vendor's need to adapt from "feature selling" as a crutch.
Domain Architect (Corporate Systems) at New Look
1yMACH alliance membership is ultimately a choice for a software supplier (and provides a useful checklist for assessment). But MACH alliance membership neither necessary or sufficient for a good solution appropriate to the business. A positive point but not a guarantee. On a related point, while single tenant may be a contra-indicator to a good architecture, it is not necessarily a restriction on good architecture (and single tenant does provide performance isolation which is helpful for guaranteeing SLA for largest customers). If the application can support multiple tenants internally at the data and configuration level (under retailer control) but is then deployed single tenant then I would suggest that is still a good architecture (assuming the other tenets of MACH are also truly present in the architecture). That said Kelly Goetsch, I agree with your vendor analysis of a year ago that you have linked in this article, for ecommerce platforms. It would be interesting to have a follow up article providing analysis for for other areas needed for a full digital solution (e.g. CRM, CDP, PIM, DAM/CMS/CDN, maybe POS, and especially Customer order focussed OMS).
Web, E-Commerce & Cloud @ Bright IT
1yThanks, I like this a lot!