Welcome Day 2 and Panel: Inside the Head of an Evaluator

Good afternoon and welcome to the second day of America's Cube, the 2024 virtual road tour. I'm Eldon Hawks, Jr. Partnership innovation specialist for Sbirsttr in the SBA's Office of Investments and Innovation. I hope that you've taken advantage of the events from today and earlier this afternoon. Yesterday, not today, because today is right now. Including meeting representatives from 10 federal agencies that offer funding through the SBIR STTR programs and meeting the road to hosts from two of our upcoming in person road tours. Be sure to review the schedule for this afternoon and tomorrow to customize your America's Seed Fund Week experience with the sessions that you care most about. Right now, we're going to take a deep dive into the evaluation process. The goal of this session is to increase your understanding of what happens once your application or proposal has been submitted to one of the SBIRS TTR programs. Now I am going to introduce or have each one of the panelists introduce themselves as we talk about what goes on inside the head of the evaluator. So these are our panelists and start off with Dr. Henry Ahn. Henry, introduce yourself, your agency and a little bit about your program.

Hi, my name is Henry Ahn, one of the program directors at NSF. Our SBIR STTR program, our annual budget is about $250 million or so. And companies applying for funding, they generally go through phase one first before applying to go phase two. When it's all said and done, it's about $2 million. For companies that are just usually just starting out, our emphasis is on startups and very early stage companies. So our hope is that by using this, this grant to advance the technology further, that companies will be able to at least have a chance to be able to continue to grow the business by either raising more money or in some cases even generating some significant revenue. Thank you.

Thank you, Haley.

Hi, I'm Haley Dowd. I am the Center Technology Transition Lead at Marshall Space Flight center at NASA. And I am also a Program specialist within the SBIR Program Office there at NASA. Our SBIR program is really focused on infusion of technologies into NASA programs. So we're trying to find technologies that will fit our needs and gaps that we have at NASA and see these grow into technologies that are flying to the moon. Our program is set up in phases. Our phase ones are $150,000 and phase twos are $850,000.

Thank you. Gayla. Jacqueline.

Good afternoon, everyone. I'm Jackie Gray. I'm the SBIR program manager at NIST and NIST also has a SBIR program that we run. We do an annual phase one where we fund eligible small businesses up to $100,000 during that six month phase one program and in the event that the any of the phase one awardees are successful with the phase two portion of the program, that is a 24 month period and we will award up to $400,000. Now at NIST we look for technical proposals that have a great idea, an innovative idea or an innovative technology that will help us solve problems or take us to another technological level in seven different research areas. Some of those areas include advanced communications, advanced manufacturing, cybersecurity, health and biologics, exploratory measurement, physical infrastructure and resilience, and cybersecurity and privacy.

Thank you, Jacqueline. Valerie.

Hi everybody. My name is Valerie Villarreal and I am the phase three Siber Sitter Commercialization lead. I'm sitting in for Candice Wright who is our SIBER Sitter program lead. She's over all of the program and I just have the phase three commercialization. Like many of the individuals on this panel, MDA Super Sitter program is also a phased program. We got phase, phase two, phase two enhancements. I specifically cover phase three which is commercialization and specifically looks at transitioning our phase twos or a phase two enhancements into a phase three which is often program funded by one of our MDA counterparts or one of our other MDA divisions that is interested in the technology. So we, we really pride ourselves on our commercialization efforts like we pride ourselves in all of our research phase ones and phase two prototyping development. So I am excited to be here again. I am not as good as Candice Wright. She's our program lead and I'm going to do my best to fill her shoes for this next hour.

Well, thank you Valerie. Don't sell yourself short. Now ladies and gentlemen, what you see here is a panel full of is distinguished knowledgeable program manager, specialist. You have years upon years represented here within the SBIR TTR world. Now I'm going to start with you Valerie. They have all heard many different things from potential applicants regarding misconceptions. Valerie, what sort of misconceptions have you heard from potential applicants for the MBA's SBIR program?

Well, some of the misconceptions is how difficult is the technical evaluation process because they don't know. They have a very skewed or cloudy view of what happens in that process because you know, that's the process that's out of their hands once they Submit their proposal, it's out of their hands, it's in the hands of other individuals that's going to direct their feet. So the misconception is that, you know what, they do have a lot of control when they submit their proposal. And when it comes to technical evaluations, if they put their best foot forward, if they are clear with the technology that they're going to present or that they have the capability to do, and that shines in their proposal, they then the technical evaluation is gonna be a lot easier and they're gonna feel confident that we're as the tech evaluator, we're gonna see exactly what they can do because they are clear.

Thank you, Valerie. Haley, what are some misconceptions about applicants applying to NASA?

So I think the biggest one, although we really like to talk about going to the moon and going to space, we, we are also looking for technologies that have applications here on Earth. NASA does a lot of work in terrestrial related topics, particularly weather. We've had topics focused on prediction of impacts of wildfires. It's a wide range of things that we're looking for. So don't just assume that because you're not directly related to an aerospace application that there may not be opportunities for you at NASA.

So nobody asks for money to figure out if the moon is really made of cheese.

Not yet.

Thank you, Haley. Henry, what are some misconceptions about applying to nsf?

As you know, NSF is probably one of the few, maybe the only agency. We're technology agnostic, so we're not looking for any specific type of technology. And as long as the company is working on developing innovative technology with commercial application, there is a fit somewhere within our program. Sometimes I get contacted by potential applicants and they ask me, well, is this the type of technology that you want to see? Well, we're not looking for any specific type of technology. So I think at this point it's probably a little more, a bit more important whether the company itself is a startup or very early stage company, because that is our focus as opposed to the type of technology. And I guess another type of misconception that I see is that sometimes companies that go through phase one, they just assume that, hey, I propose to do this project. As long as I complete this project and successfully meet all the, all the milestones successfully, I'll be moving on to phase two. And that generally is not the case. Our phase two applicants are expected to be much further along, not just on the technical side, but on the business side as well. Because in the end we're supporting for profit companies and we want companies to be able to continue to grow the business beyond the grant money that we can provide. So I think it's sometimes companies do have to I mean, yes, as a startup with just one or two people, sometimes it's very difficult to do all the work that they're proposed to that they're supposed to do, technical work, and in addition to that, be out there talking to potential investors, potential strategic partners, and potential customers. All of this is it's a lot of work. But just don't assume that just because you do the technical work that you'll be transitioning to a phase two. And we do emphasize this quite a bit with all of our grantees.

Thank you. Jackie. Misconceptions about applying to nist?

Sure, Alvin, I can think of a few. One misconception that I've noticed that some applicants have concerns about is that once they submit a proposal, say for phase one, and if they don't win, they can't resubmit. No, that is further from the truth in the I always say in the first you don't succeed. Try, try again. You when applicants submit a proposal, a technical proposal, and they're not successful, they have the opportunity of getting a debrief where they can take notes and learn how to beef up those areas where they fell short. And when that solicitation opens again, I encourage them to please send in the application again. I've seen it where not only the first time that an applicant does not get a win, they try a second time, use the debrief, gain all the knowledge, resubmit, didn't win again, third time I've seen it where they actually use all of the information, beef up the technical proposal, and some of them have been successful. Another misconception is oh, NIST is just technical. They just deal with measurement science and and that deep down technical stuff. NIST is involved in a variety of research and development activities. As I mentioned previously, we deal with physical infrastructure and resilience and that entails working with communities. We may be working on projects where there's an advanced warning system put out to communities that notifies residents of impending or disasters of some type that is affecting their neighborhood. So we're just not involved in the deep down measurement science, but we are involved with different research areas across the board. And if you'd like to find out more about those different areas, topic areas that we're doing research in and the different programs and projects, I invite you to go to our homepage which is www.nist.gov. and in the middle of that homepage you'll see all of those topics, all of those research areas. Please click on them and all of the programs and projects will fall out and you'll see a cross section of all the exciting things research areas that we're working on at nist.

Thank you, Jackie. Jackie, real quick, does NIST offer grants or contracts?

We offer grants and the grants are cooperative agreements. They are not contracts. Because what contracts are governed by the Federal Acquisition Regulations, the far. So the grants operate under a separate state set of rules and procedures, not the far.

Thank you, Valerie. Real quick, grants or contract.

We fall under the far. So we are contracts.

Thank you, Haley. Grants of contracts.

We have contracts as well.

Thank you. And Henry, Grants of contracts.

Hence.

Okay, so I want to put that out there because what you're going to see as we go forward, you're going to see some differences in the evaluation process between grants and contracts. I want you to take notice of which agencies do grants, at least on this panel, and which agencies do contracts. Now, I'm going to you, Henry, because you brought up something during your earlier words about innovation and specifically talking about phase one, that phase one application, because this definition, all these what needs to be illustrated is going to be a bit different. How does one define innovation in that phase one application for this?

Yes, I think in generally, I mean when a company is working on developing innovative technologies, so the technology itself, there has to be some components that may be considered innovative, novel enough to in some cases to be to be patented. And we do see that quite a bit. But I think in most of the cases when companies approach us, they do have to come in with some evidence that the technology has the potential to work. And when we say innovative, it generally means not doesn't does not mean just putting together something that you already developed. Say you've already developed this technology for one application, you want to use it for another application. Maybe it's just an incremental improvement to the existing product. Maybe it's just an existing product that existing components that you just put together as a device may be just seen as incremental as opposed to innovative. But the way that you put together was never been done before. Then perhaps there is innovative component. So I think depending on what the company is trying to do, that term may be a little different. I handle the biomedical technologies topic within our program and for my topic, that generally means that something that companies will be seeking patent protections for. But for topics that are under software, for example, that another program director within NSF handles a patent may not necessarily be all that important and may not even indicate that the innovation that it's sufficiently innovated. So depending on the type of technology, because we handle a broader topic, I guess all different types of innovative technologies, depending on the type of technology that you offer, it may be a little different. So one thing that I would strongly encourage you to do is to we have a project pitch step. So if you come to our website, submit a project pitch, we'll be able to review and get back to you. And if the innovative component isn't there for that particular topic, you'll at least know about it. And if you do have questions about it, you'll know exactly who to reach out to to have additional discussions on what we mean by innovation. And so that's what I would strongly suggest that you do. Just submit a project pitch first and you can have additional discussions with the appropriate program director in terms of if it's seen as lacking innovation, you'll be able to hear directly from the program director who handles that topic on why it's lacking innovation versus having innovation within the technology.

Thank you, Jackie. How does someone articulate innovation for Phase one Finist?

They would articulate that by in their technical proposal showing that they have done extensive research in the area. They've done their market research to first and foremost to see if there's a need for it. And they have also been checking in with the investment community to see in the event that their idea does have some merit, that the investment community will be receptive to providing seed capital to move this thing forward. As Henry mentioned, we look for innovative technologies and ideas that are not commercialized. NIST will not entertain a technical proposal where the firm already has that product or service out in the marketplace. So the bottom line is NIST is not going to fund you an award for technology that you've developed and to have it sit on the shelf. It's all about whether that technological idea has potential for commercialization.

Thank you. Jacqueline Haley, same question. Innovation. How does someone articulate innovation for NASA Phase one?

Yeah. So for NASA, we're looking for novel technologies that meet the needs that are put forward in the solicitation that's put out annually. So our solicitation is actually out right now. Within that there are subtopics that reference the needs or gaps that NASA has currently that they're looking for solutions for. So when your proposal comes in, we'll be looking at whether or not the proposed technology meets those needs, is technically Strong. And there's confidence that what you develop will be able to be used for whatever gap there is in the technology currently. So we are looking at the ability to commercialize, but we're also looking at the ability for NASA to use that technology.

Thank you. Valerie, what does innovation look like for phase one with NDA?

Okay, so meeting the phase three experts, I'm going to dive into phase one as best as I can.

No problem.

So the Missile Defense Agency has 15 research area leads, research areas that we look at and we live in in a daily basis. MDA is set up within these 15 areas that comprise our organization. And when those solicitations come out, our 15 areas are almost always represented. Now to be a successful phase one, you got to know your area that you fall into. In regards to our 15 research areas, some of those comprise into test instrumentation. Some of them are very specific to thaad, a terminal high altitude program. Some of them are specific to our ground weapons or ground mid course defense programs. You have to know where you fall into these research areas that we are requesting information from that fill in our gaps, our need gaps. So you gotta know who you are in order to be successful in our phase one. You got it. Like Jackie had mentioned, you know, do your market research. Well, for an MDA phase one, you got to do your market research, not necessarily to know if the innovation is needed, but you got to do your market research for our phase one to know who your competition is and get an idea of who you're competing with. That market research is also going to show you, if you read between the fine lines and you do a good thorough market research, which by the way takes time. You can't do market research in a day. You gotta, you gotta be doing this ongoing because you know who you are and you have an idea of where you fit into. You're doing your market research rather routinely, so you have an idea of who you are and what your competition is. So for an MDA successful phase one novel idea, of course. Can it be commercialized? Can it reach my lane in a phase three commercialization? Yes. Well, that would be the goal, right? You don't want to be in that valley of death. Get to a phase one and you're good enough to get to a phase two and then you sink into the valley of death for a variety of reasons like funding or, you know, the reality is just doesn't match your technology yet or vice versa. But you got to know who you are, you got to know where you fit into within the mda. Fifteen areas of our research that we look for.

Thank you, Valerie. And Valerie, you must have known my next part was going to talk about commercialization. I believe we can start with you, but you've already answered my question.

So.

I'm going to go to Jackie. Jackie, talk about the importance of articulating commercialization in a Phase one application for nist.

Sure. Alvin. Commercialization is very important because as the panel, most of us have said previously and especially at nist, we are looking for innovative ideas or technological solutions that will help NIST solve problems across multiple areas of research. And it definitely we look for it to have potential for commercialization. So not only is the commercialization going to help that small business, but it's going to help NIST solve problems and it's going to benefit the community, the national economy as a whole. This is why the federal government allocates money to programs like SBIR and STTR for it to benefit the national economy. Small business, large businesses, this is how we all come together collectively to drive innovation in technology.

Thank you, Jackie. Haley, could you talk about commercialization, how important it is to articulate it in Phase one?

Absolutely. During the Phase one review process, there are four factors that are evaluated and one of those is commercial potential. So it is important to include an understanding of who potential customers could be. What's the potential value of your technology. But I will say that during phase one, the technical review portion of the proposal is scored at a higher value. Phase two is where commercialization really comes into play.

Okay, Right, exactly. That's why I want to make sure, like because people ask all the times about commercialization. Phase one, what's important? How important is it, how to articulate it? I know that for everybody across the world, Phase two is where really that commercialization aspect is really looked at. But I want people to understand that it is going to be a portion of your Phase one application. Henry, talk about articulating commercialization potential in Phase one.

Yeah. For nsf, I would say the commercial section is just as important as the technical section. As most of you probably know, NSF does not have any internal research. We're not looking to bring any technology in. So it's all, it's great if the company is working on something very interesting and add specific knowledge to the field. But we're supporting this for profit company and our money really doesn't mean anything unless the company is able to continue to grow as for profit businesses. So for that reason, we do expect the Phase one proposal to have significant amount of commercial info Does a company understand the market potential, who the competitors are, significant competitive advantage in some cases under my topic, under biomedical technologies, they have to have some good understanding of what the regulatory path to market might be or any aspect of the intellectual property property patented. If not, how will they protect it? How will they compete and continue to grow the business? And all these things are extremely important because that sometimes shapes the way that a company proposes a phase one project. A company may be planning to license the technology to another company, for example, as opposed to just commercializing on their own and market the product themselves. Then maybe they have some understanding of what that commercial partner may want to see. And sometimes that gets incorporated into the technical milestones. Depending on how far they need to bring the technology, develop the technology, the amount of money that they need will be a little different. So they have to start thinking about, well, when I go and start making these investor pitches, how much do I have to request, how far do I have to bring this? And having some either strategics or investors, their interest in helping them get to that point of being able to continue to grow the business, those things will always help strengthen the proposal.

Thank you, Henry. Now I want to kind of move toward the evaluation process. I'm going to start with you, Henry, first. Who are you evaluators like, who's actually reviewing these proposals to make the decision?

We have people that are considered technical reviewers who tend to be faculty members at a university somewhere within the country working on technologies that may be in a similar space. And we have people that are considered commercial reviewers. And these commercial reviewers may be, sometimes they're serial entrepreneurs who are familiar with the market sector and sometimes they're early stage investors. We have some clinicians in my case, sometimes they bring, they come in to talk about the need for a particular type of technology. Sometimes we have some consultants, such as regulatory consultants, or sometimes just consultants working with small businesses. So people that may have some background other than just the technical side. Most of our commercial reviewers tend to have enough technical background to comment on the technical side as well. But they're generally brought in because of their expertise or experience in the non technical side. So the goal with our review process is to have these people with different backgrounds and different expertise to review the proposal, submit their reviews and get together in a panel and we'll talk about all the strengths and weaknesses. And sometimes we have a technical reviewer who will rate a proposal very high, but a business reviewer may rate it really low. So that's when the discussion becomes a little more interesting. I think we have to talk about, well, can the company get to that next point where they can continue to grow? Is this deficiency on the technical, on the business side, is this a critical flaw or will it take a long time for them to address? If that is the case, then the company will most likely be declined. And the company can always address those concerns and come back and submit their proposal again. As Jackie said earlier, companies are always welcome to come back and update their full proposal once they're able to address those concerns and go through the review process again. And that certainly is the case for NSF as well. So in terms of reviewers. Yes. So those are the type of reviewers that we will bring in. And I think one other thing that I should probably mention is that each reviewer will rate a proposal before the panel gets together. They'll review. They'll rate it as good, fair, poor, very good, excellent. But those ratings don't necessarily translate to whether a proposal gets recommended or not, because as a panel, we'll discuss and as a group we'll decide, well, is this a strong proposal, is this a competitive proposal or not? So we don't have a cutoff system where one reviewer will dictate, I guess, whether a proposal will be meeting the cutoff line or not. As a group, we'll decide and decide, discuss and decide whether the proposal is competitive enough to be considered for funding.

Thank you. I move on to Haley, but I want to make this point. I hope everybody's picking this up because we have a question about resubmitting, because both Jackie and Henry have both talked about resubmitting. One of the things you see with the granting agencies, that's Keith and this is and NSF they talked about recently. Typically, granting agents have topic areas that are very broad. They're about funding science, whereas the contracting agencies, they're looking for something very specific. They're looking to fund a specific technology to meet their needs. And correct me if I'm wrong, both Valerie and Haley, those needs change or can change from year to year. So that's why you hear that discussion on one side of things is, oh, you can resubmit as many times you want. Others, no, we just want it. If we don't get it this year from you, then we have something totally different next year. So that's why you're seeing that difference. Now, Haley, who reviews the applications for NASA.

So for NASA, we have subject matter experts that are reviewing the proposals, which means these are scientists, engineers, technologists that are experts in the given technology area, and they are the Potential customer for you. They are the ones that understand NASA's needs and understand where your technology could fit within our missions. So they are looking for technologies that fit those needs, but they're experts in those areas so they can understand all of the technical jargon that's included in your proposal.

Now, Hayley, are these internal or external to NASA?

These are typically internal. They can occasionally be external, but they're usually internal.

Thank you, Valerie. Who reviews the application for mda?

Yeah, so MDA is strictly internal. Due to the sensitivity of the Missile Defense Agency, we've got subject matter experts just like NASA does. And they're engineers, physicists, senior engineers. Just depending on the proposal, the technology area that it falls into, the research area that it falls into, it's either punted over to very, very specific engineer or physicist that is looking for that particular technology to fill a gap, or it's going to be a subject matter expert that is general, depending on the technology. So yeah, so we are subject matter experts, we are internal and it can go to a general subject matter experts, depending on the technology and the need and so forth, or it can go to a very, very specific research area, lead engineer, physicist, that is going to review that because they are filling a need.

Thank you, Valerie. Jackie, let's talk about who reviews the applications for nist.

Sure. At nist, those that review our technical applications and proposals. They are comprised of scientists, engineers and technologists, including subject matter experts at NIST and this group, they are NIST federal employees. However, we also employ a cadre of what we call domestic guest researchers. These are consultant type personnel who are are working at NIST and they are invited to review proposals according to the specialty of that type, particular topic of that proposal. Of course, the domestic guest researchers, they do have to sign a special confidentiality non disclosure agreement. And we also, in addition to domestic guest researchers, we have what we call foreign or international guest researchers. That group of consultants are not allowed or not invited to review technical proposals at nist.

Thank you, Jackie. I hope you see that regardless of the agency, whether it's contracting or granting, well, contracting, that all of the ladies here, they all review in house. Because typically those people that review are part of the process that designs the actual solicitations. Whereas granting, you can have a mix of in house as well as external. So a lot of it's going to depend on the agency and what sort of technology the agency is looking for. So I want everyone to take that into account. Let's talk about, we're going to talk about the technical portion. But one of the most important parts, and it's something that everybody's brought up, is the team. How important the hidden grand salt was, how important is it to really put your team together and to express your capabilities of your team in that phase one application?

Is that for.

I'm starting to start with Henry.

Okay, so NSF's emphasis is on startups. So most of the companies that we see, they're just getting started. Some are just newly formed, some are just a few years old. So for that reason, we generally don't see a full set of team when they submit a phase one proposal. They tend to be people, inventors or people that somehow got involved with the technology early on. But we do believe that having a strong team is very important. So instead of, I guess, within the team, if a company that's just getting started with one or two people, if they can just work on having an advisory board where these people can fill the gap, managerial gap that may exist within the company until the company is able to bring them in, that will be sufficient. But the team is definitely important no matter who you talk to. I mean, the team will always be important. Whether it's US investors, especially angel investors, it'll be really important. But because most of the companies that we see can't show that they have a full team, just please have an advisory board of some sort you can point to and say, hey, we have these experts on our advisory board and they will be providing us with guidance. We meet with them periodically and they provide us with guidance until we're able to bring those experts within the team.

Thank you, Valerie. I'm going to jump down to you, talk about how important it is to put a great team together for your phase one and just in general for your proposal.

Yeah. So it's pretty darn important, you know, for any of our phases, it's pretty darn important to highlight your team in the expertise that they are going to bring to the project. Now, the phase ones are, you know, the concept portion of the whole super sitter program. So you should take the opportunity to highlight the team members who are going to be developing that concept or who have developed the concept. Remember, the Missile Defense Agency is a little bit different because we've got 15 areas that we are researching that we need to fill a gap. So that concept and that concept team is really important because they have to have the expertise and they got to show their expertise to help us fill that research area gap. So team very important. Absolutely. Highlight your team, highlight their experience experience highlight their expertise.

Thank you, Robert. Jackie.

Sure. The importance of putting in your proposal that you have a really strong, well rounded team is paramount. However, as Henry mentioned that lots of times a small business only may have one or two team members on board at that time. And NIST recognizes that in their phase one because the importance of the experience of the team, investigators and support staff, that is the, we allow the lowest amount of points for that because we do understand when they're first starting out, they may not have that broad brush of expertise. However, when a small business recognizes that they have the technological idea, but they want to round it out and make it a really robust proposal, they may include a consultant or a subcontractor to help round out any areas that they may fall a little short on in comparison or trying to meet the multiple evaluation criteria. So that is a consideration that the team could consider if they think that it may add or augment. Their technical proposal is to use consultants or subcontractors.

Thank you, Jackie. Haley, talk about how important it is for the team to be expressed. And that's.

It's very important. And what that comes down to really is you want to make sure that the reviewers are confident that the team can perform the work that's being described in the proposal. Right. The subject matter experts are aware of what it's going to take to do that work and they're going to look at the team and see does this team have the expertise and capabilities to do that. And if they don't, they're going to highlight that as a weakness in their review. So it's very important to have a strong team and have that well described in your proposal.

Now, is there, I'm keeping with you, Haley. Is it such thing as being too technical in your proposal?

No. So the people that will be reviewing it will understand what you're talking about in your proposal. I don't think you can be too technical. I would make sure that you are describing all of the components that are requested in this solicitation and making sure that your page limits are within those. So be careful about being so verbose that you're going over your page limits, but I don't think it's possible to be too technical.

Thank you. Henry, same question.

Yeah, I mean, you just can't be too technical, but you do have that page limit. So you have to be careful about how much information to include. And we need to have enough details to be able to for our reviewers to evaluate. And I would say yes, I guess it may be considered Too technical. If the proposal has nothing but technical information, almost 15 pages of technical information, but maybe then you're being a little too technical. You won't have enough room to have enough commercial information there to come out of the review panel positively. So just be mindful of the page limit and make sure to include sufficient details to be able to, for us to be able to evaluate. And don't worry about whether something is too technical. One other thing that I would say is just also be careful about using too many acronyms. I know being government employees, we're used to many acronyms, but when we see proposals with a bunch of acronyms in there without having to define them, our reviewers definitely do get confused. And we even, you know, we, we get confused as well as we go through the review to the proposal. So make sure to define, define them, but include sufficient details for us to be able to, to, for us to be able to evaluate and clarify whenever you can so that it's a little easier for us to understand in terms of understanding the acronyms and others.

Thank you, Jackie. This thing is too technical.

Yeah. I would like to piggyback on some of the points my fellow panelists brought out. I have seen some proposals that I would deem too technical because what, as Henry said, they're loaded with acronyms that aren't defined. And then I've seen some technical proposals. Even though we have a 15 page limit, it's all text. Try to put some illustrations in there to drive home what you're trying to convey and that will break up the monotony. It won't put the reader, the evaluator, to sleep. And try not to lean to. Try not to lean too heavily on formulas and equations because that's when you're really getting down into the weeds of being too technical. When a paragraph or two of plain English language that the layman can understand would really drive home the point.

Thank you, Jackie. Valerie, same question. Is it such things being too technical?

Well, I think everything that my fellow panelists have mentioned is very important and they've captured everything. I do believe that you could be too, too, too technical and not highlight illustrations and not utilize illustrations. I think you could bore a technical evaluator to death if you are heavy in formulas. So yeah, you can be too technical. But I think the most important thing with the Missile Defense Agency when it comes to technical evaluations and technology and being technical, being clear and concise. Can you say what you need to say within a paragraph versus 45 paragraphs because you have lost A technical evaluator, no matter how good they are or how high of a level of an expert they are pertaining to your area that you've sent your proposal for, did you lose them? So clear and concise is very, very important, I think, for any proposal. Can it be too technical? Well, if you're too wordy, you're going to lose them. If you don't give enough words, you've lost them because they're confused. So finding that balance. Be clear and concise. Be clear and concise for the Missile Defense Agency goes a long way.

I think everybody brought up very good points as far as to really address that. Too technical. Spot on everybody. Now I want to ask this question, okay? So we know what the success rate is for phase one. You know, about 15% across the board or so. And I'm going to start with you, Valerie. You get rejected, what kind of feedback do you receive from the agency?

So you have to ask for feedback, okay? And you got to be open and receptive to the feedback. Now, there are certain things that the technical experts or subject matter experts can say and can't say, okay? Because they've got to be within the bounds and be sensitive to the Missile Defense Agency boundaries and confines. So you have to ask for the feedback. We give you general feedback on why it wasn't selected, but you really have to ask for it. You have to go out and seek.

It.

And most importantly, be receptive to the feedback that is given to you. It's hard to hear, right? When you get rejected, depending on whatever it is, it's hard to hear. And a small business owner providing technology is very proud of their technology. They have to be. They have to be confident, they have to be proud. But when they don't get selected for funding from a phase one to a phase two, they have to understand why. So be open to it.

Thank you. Jackie, Real quick, what kind of feedback can an applicant receive if they do not get selected?

Sure. As Valerie says, at nist, the applicant, unsuccessful applicant, has to request a debrief. And several years ago NIST would mail out the results of their evaluation. We don't do that anymore because, you know, it got pretty contentious across some cases. So now we conduct a 101 and we go through the different evaluation criteria and we go over the scores, the ones that fell short. We let them know, do a summary of what the evaluator said, why it was low, and that way the applicant, the unsuccessful applicant goes away with real time information on what went wrong and how to correct it. In the future because as I said earlier, our topics don't change much from year to year. And so they get an opportunity to dust off their technical proposal and resubmit again. And so this is excellent feedback doing our one on one. They get to ask me any question because it's all live. They can get written reports. I give them an opportunity, if I didn't say something clearly, they can ask me again. So it's that their one time shot to gain all of that information. Effective and constructive feedback, dust off that proposal and again refine it and resubmit for the following phase one annual Phase one solicitation.

Thank you. Jackie Haley. What kind of feedback can when expect.

For phase one firms get written feedback? So that's after the selection letters have gone out. They'll get written feedback that talks about the strengths and the weaknesses related to the different evaluation factors.

Thank you, Henry. Same question.

Phase one companies, there will be a minimum of three individual reviews, reviews from individual reviewers and most of the time there will also be a panel summary based on the panel discussion. And all the individual reviews, they will be there verbatim, minus the identity of the reviewer. Of course. It's provided anonymously, but everything is there. And sometimes I emphasize with my reviewers that this is our feedback to our applicants. So even if the company decides not to come back and submit a proposal again, we hope that this feedback will be helpful to them in some way. So some of my reviewers actually take that to heart and they write a lot of things in there, a lot of information. So the minimum of 3 individual reviews verbatim and plus the panel summary, those documents companies can get to when they log into research.gov, they can get to these documents and they can review them. They can also request a debriefing call and we can go over some things that may not have been clear. We can't really. Sometimes when I hold these calls, sometimes they just want to go to a specific comment within one individual reviewer and say, hey, this reviewer didn't get what we were trying to do and they're just very upset about one small comment from one reviewer. And I often have to kind of bring them back and say, hey, if it's not found in the panel summary, let's not worry about this, but let's talk about some of the things that were not clear to you within the panel summary and what you might be able to do to strengthen your position or strengthen your proposal so that you have a better chance of getting funded next time around when if they decide to come back and submit it again. So those are the feedback that they will receive, those reviews, individual reviews, panel summary, as well as a debriefing call if they request it.

Thank you, Henry. Now, really quickly, just few seconds, Henry, then Haley, then Valerie, then Jackie. Just some parting words for applicants to make a good impression on their proposal.

I think the most important thing is to follow the solicitation guideline very carefully. I can't emphasize this enough and I often see proposals that are just purely recycled proposals from NIH and our reviewers. Many of them do serve in other agencies and they'll immediately know the format and say, hey, this is not really an NSF proposal. If they don't take the time to go through the solicitation and provide the type of information that we require them, then it's probably not going to be a strong proposal. So follow the solicitation guideline very carefully. And up until the time of full proposal submission, they can always reach out and ask questions if they have any. Once you submit a full proposal, there will be a period of silence until we go through the full review process. But you can ask questions. And we also have Q and A webinars just about. I think it used to be once a week. Now we've kind of separated them out. So it's. We alternate between getting to know NSF SBIR program versus preparing your phase one proposal. So those webinars will alternate on weekly basis. So every week we have something. You can go to our website and sign up our website. The easiest way to get there is steedfund.nsf.gov if you go to that website, on the upper right hand corner, there's a link that says how it works. You can get to the project pitch portal there. And in the middle under the Resources tab, you can get to the events where you can sign up for these webinars.

Thank you, Haley. Real quick parting words.

Yeah. So I would echo Henry's information about the solicitation, read the solicitation and know it, and submit your proposal according to that process with all of the content that are required in there. Also, NASA's a little bit different in that we do release the solicitation and when that is open, you cannot communicate with NASA subject matter experts about the solicitation. So if you want to talk to someone about specific topics, do that before the solicitation is released. And that's typically released in January. So look at the fall as your opportunity to talk to subject matter experts and reach out at that time.

Thank you, Haley. Jackie, what parting word?

Parting words are three things. Please read the solicitation. As my fellow panelists have stressed, have a second party look at your proposal. Because you've been dealing with it for so long, you may be overlooking some things that another party may zone in and see what the shortfalls, what the errors are right off the bat. Third thing is please don't wait until the last minute to submit your proposal. Proposal Collecting services. They are not infallible. They have glitches. And your proposal could get hung up at the last minute and when the clock strikes midnight. And if it's not through, all that hard work you put in is all for naught. So those are my three top parting words and actions that applicants need to take heed of.

Thank you Valerie. Parting words.

Well, I'm going to say ditto to what Henry and Haley and definitely what Jackie said about don't wait till the very end because of technical issues. Because that's happened, right? That's probably happened to a lot of people. So with the Missile Defense Agency, be concise, be clear. Follow the solicitation guidelines, meet the requirements that they're asking for you for from you in the solicitation. It's all really important. Be clear, concise, be happy about doing it and don't wait till the last minute.

Thank you very much Henry, Haley, Jackie and Valerie. Thank you so much for your time and thank you for sharing your expertise with this humongous crowd we have here today.

You're quite welcome.

Now. Thank you. Yes, you, you, all of you wonderful individuals that are watching us and learning all this information. For joining us today, I invite you to browse the schedule to customize your America Seed Fund Week experience with the sessions that you care most about. Beginning at 2pm Eastern you have the opportunity to interact with representatives from eight federal agencies and concurrent broadcast sessions or breakout sessions, excuse me, focused on AI ML, space transport flight, the NSF project pitch and phase three with the DoD tomorrow. There's also a special session offered at 11am about preparing for your one on one meetings at the in person road tours, the chance to meet host on two of our upcoming in person road tours as well as many opportunities to interact with agency representatives. Thank you for taking the time to be here. Thank you for coming to learn and also remember, learning is half the battle.