Learnings From Project Caspian: Our Purpose-Built Security Data Platform
By Jericho Cain, Senior Staff Data Scientist, Hayden Beadles, Senior Security Machine Learning Engineer, Bharat Gamini, Senior Security Data Engineer, and Karthik Venkatesan, Senior Manager of Security Software Engineering
We take a thoughtful, integrated approach to security, working together to keep Adobe and our customers safe. The security team brings clarity to cybersecurity challenges, supports our customers, and partners across communities to build trust and transparency across the industry. We are continuously learning from our global teams and community partners to help stay up to date on the latest trends.
Typical security tools, like EDR, can generate multiple terabytes of data per day, which must be effectively and quickly analyzed to find useful nuggets of information. When we began examining how to more effectively implement early detection, we found available off-the-shelf tools could not adequately capture, collate, and analyze data at the scale we were seeking. Thus, we began to research developing our own security data platform.
We started researching available solutions to help us land on a reasonable path to organizing and structuring security-related datasets that could enable our teams to conduct more effective threat detection. Through this research we found that available tools lacked necessary performance, data management, scalability, and security controls we required to help address the problem at our scale. The result of this research is the creation of a purpose-built security data platform we call “Project Caspian.” Project Caspian is now used within our security teams helping them more quickly and efficiently analyze the vast volumes of security data we collect to find actionable items that help us drive better security decisions.
To support the broader security community, we are publishing a duo of core research papers from the project. For in-depth insights into Project Caspian, we’ve published a paper entitled, “Accelerating Security Analytics with a Purpose-Built Security Data Platform.” We’ve also authored a second research paper entitled, “Anomaly Detection in Okta Logs Using Autoencoders,” that delves into a specific use case, showcasing how Project Caspian enables more purposeful and efficient machine learning and deep learning techniques to help proactively detect threats. This paper also discusses the models we used, techniques at play, and useful outcomes that can be achieved leveraging security data platform technology.
We invite you to download both research papers to dive deeper into our research and approach. Adobe believes that Project Caspian can open doors to numerous possibilities for our security teams to make more effective use of data that improves our ability to respond to ever-evolving threats.
Watch this space where we will share additional research on our latest security projects.
What’s on Your Mind? We Want to Hear from You!
Your opinion matters to us. Help shape the future of our blog by sharing your ideas and preferences. Click the link below to take a quick survey and tell us what you’d like to read about next.
