Prompt Security wants to make GenAI safe for the enterprise

Businesses are moving faster than ever to use generative AI and bring it to both their employees and users. Moving fast and security don’t always go hand-in-hand, though, so it’s only now that many businesses are waking up to the potential security concerns related to using generative AI. Israel-based Prompt Security wants to help organizations ensure that their employees aren’t leaking data to GenAI tools — including those not officially endorsed by the company’s IT teams (think ‘shadow AI’) — while also helping them secure their own customer-facing GenAI-enabled application.

The company is coming out of stealth today and announcing a $5 million seed round led by Hetz Ventures, with participation from Four Rivers and a number of angel investors in this space, including CISOs at Airbnb, Elastic and Dolby.

Prompt Security was founded by Itamar Golan (CEO) and Lior Drihem (CTO), who both previously worked at Check Point and Orca Security. “I always knew that I wanted to found my own company. My two biggest passions were AI and security, but five years ago, I knew that the market was not mature enough. So I waited for the right moment,” Golan told me.

Then, about 18 months ago, his team at Orca started looking into implementing an AI-based remediation feature, which he ended up building together with Drihem. That’s what inspired the team to start this new company. “We saw the different backdoors that we were leaving through the development and we looked at each other and we said: there is a new attack surface due to these new architectures and new frameworks — things like prompt injection are suddenly impossible — and then we understood that a new market is about to thrive.”

Golan is also part of the OWASP Top 10 initiative for enhancing the security of LLM applications.

Part of the problem for CISOs today is that they don’t have a good understanding of which tools their users are even accessing from within their networks, Golan explained. That also leaves the door open for employees to, for example, share personally identifiable information with these services. This kind of data leakage is problematic in and of itself, but it could also open the company up to regulatory issues. From the user perspective, Prompt Security’s product for ensuring that this doesn’t happen is a browser extension and IDE plugin. The company’s tools automatically detect patterns related to GenAI usage and then layers an enforcement policy on top of that. All of this, the company says, only introduces minimal latency to the user experience.

Meanwhile, on the application side, GenAI chat apps open up a new attack surface through prompt injections and jailbreaks, which may even go as far as allowing an attacker to run unauthorized code inside a company’s network.

Golan stressed that the company is trying to build an entire platform here by covering various aspects of an organization’s GenAI usage. “We are trying to build a one-stop solution for GenAI security. We are covering both the employees’ usage — the corporate InfoSec side — and the application security side.”

Only a few months ago, Golan told me, many CISOs were still either unaware or unconcerned about GenAI security issues, but that’s changing rapidly now that they are seeing an increase in shadow AI usage.

Over time, the company plans to launch more services that help its customers increase their GenAI security posture. Currently, Prompt Security’s focus is mostly on securing applications in production, but the team plans to increasingly focus on helping developer secure their applications during the development and testing phases. “I think that we need to be very focused at the moment on the current needs around the Gen AI: visibility and data privacy in real time,” Golan said when I asked him about the company’s expansion plans. “Later on, we will expand with more money and more customer feedback — and I would say that this market is so crazy, what I’m telling you now probably wouldn’t be relevant next year.”