Most founders evaluate agencies based on their portfolio, their pitch, and how responsive they are in the sales process. These are the three worst proxies for what actually matters.
An agency can have a beautiful portfolio of products they shipped and then walked away from — products the client is now struggling to maintain. Their pitch can be polished because they've given it a hundred times, not because they actually understand your problem. And they're responsive in the sales process because that's the part where they need to close you.
The things that actually determine whether an agency engagement goes well are not visible in the sales process. You have to look for them deliberately.
What you're actually evaluating
When you hire an agency to build your product, you're making three separate bets:
- That their technical judgment is good enough to make the architectural decisions that will define your product for the next two years
- That their delivery processes are reliable enough that what they promised will actually get built
- That the relationship will survive the inevitable moments of disagreement, delay, or discovery about what "done" actually means
Most founders focus almost entirely on the third. The sales process is designed to build that confidence. The first two — which matter much more — require active investigation.
Questions that actually reveal technical quality
"Can we see the codebase of one of your past projects?" You won't always get a yes to this, but how they respond tells you something. The best agencies will offer to walk you through a sanitised version or show you code from an internal project. If the immediate response is defensiveness, that's a signal.
"Tell me about a time a project went over scope or timeline, and walk me through exactly what happened." Every agency has had this happen. The question is whether they'll be honest about it and whether they've learned from it. An agency that claims every project went perfectly is either not telling you the truth or hasn't done enough work to encounter the complications that are a normal part of software development.
"Who will actually be working on my project? Can I meet them?" Sales-to-delivery bait-and-switch is one of the most common problems in software agency engagements. You meet the senior team in the pitch. The work is done by juniors you haven't met. Ask explicitly who will be assigned to your project, ask to meet them before signing, and ask whether this team has availability for the duration of your engagement.
"What happens to the codebase after you deliver?" This is the question most founders forget to ask. What documentation will exist? What does onboarding a new developer look like? What does post-launch support look like? If the agency's answer is vague — "we'll document things" — push for specifics. The agencies that think seriously about post-delivery state are the ones that understand they're building something that will outlast the engagement.
"How do you handle technical debt during a build?" A thoughtful answer to this question is a positive signal. The agency is acknowledging that tradeoffs happen and demonstrating that they have a process for making them transparently. An answer that suggests technical debt isn't a concern — or worse, doesn't seem to understand the question — is a red flag.
The contract red flags
The contract often tells you more than the pitch. Things to watch for:
Deliverables defined by features, not outcomes. If the contract specifies what will be built but not what "working correctly" means, you have no recourse when you receive something that technically matches the spec but doesn't solve your problem.
No code review or quality milestones. The engagement structure should include checkpoints where you (or someone technical on your behalf) can review the work in progress. An agency that resists mid-project review is an agency that doesn't want their work examined until it's done — which is when it's most expensive to fix.
Intellectual property ambiguity. Everything built during the engagement should be yours. Read the IP clauses carefully. Agencies occasionally retain rights to components or frameworks they consider "proprietary" — understand exactly what you will and will not own.
Payment structures that front-load risk to you. Milestone-based payment structures that align the agency's payment with delivery of working software protect you. Large upfront payments before any code is written do not.
What to do if you don't have technical expertise
If you can't evaluate the technical quality of an agency's work yourself — which is true for most non-technical founders — you have two options.
The first is to bring in a technical advisor specifically for the evaluation process. This can be as simple as paying a senior engineer for five hours to review the agency's past codebase, ask the technical questions in the evaluation call, and give you an honest read on the quality you'd likely receive.
The second is to run a paid technical pilot before signing a full engagement. Ask the agency to complete a small, well-defined piece of work — one or two weeks, clearly scoped — before you commit to the full project. This gives you real data on their code quality, their communication style, and their delivery process before you've committed significant capital.
Both of these options are worth the cost. The alternative — signing a $60,000 engagement without meaningful technical evaluation — is the setup for the scenario that produces $80,000 rebuilds twelve months later.
The right relationship with an agency
Even the best agency is not a substitute for technical leadership on your side. Agencies build what you tell them to build, within the scope you define, to the specification you set. If the specification is wrong, or the scope is misaligned with what your product actually needs, the best agency in the world will deliver the wrong thing.
Technical leadership — whether fractional or full-time — is what ensures the specification is right, the scope is grounded in your product strategy, and the agency's work is being reviewed with genuine technical judgment throughout the engagement.
Agencies are a tool. Used well, they can accelerate delivery significantly. Used without proper oversight, they're a very expensive way to learn what you should have built differently.
Foundry works with founders to provide the technical oversight that makes agency engagements work — and to rescue the ones that didn't. Book a free intro call and let's talk about where you are.