Junior DataOps Engineer (#1854)

We are looking for an outstanding Junior DataOps Engineer for Recorded Future. Recorded Future collects intelligence from many sources in “threat actor space.” This data collection includes terabytes of data dumps that are shared and trafficked underground, as well as hundreds of dark web forums, markets for stolen goods, and ransomware sites. The Identity team builds the data collection and analytics that transform this mass of raw data into structured intelligence information that our customers use to defend their people and networks. You will be responsible for data collection and operational data engineering on data dumps and “malware logs” from infected computers.

Responsibilities:
● Collect raw data from the threat actor environment on a daily basis
● Supervise automatic processing of this structured raw data. Investigate and
resolve data processing problems. Fix and improve our data processing code so it will automatically handle similar data scenarios.
● Process and index massive stolen credential data dumps. Assess the security value of data in a dump file, assess the technical format (SQL dump, structured text data), decide how to process the data to extract the key security information, and execute on the tasks of processing and indexing. Maintain existing ETL AWS
pipelines for credential ingestion.
● Develop, debug, and maintain data collection pipelines, manage APIs (Telegram, Mega.nz).
● Collaborate with other teams on data gathering.
● Document data collection processes.
● Utilize and manage Elasticsearch for data indexing and search.

Requirements:
● Very flexible and outcome-oriented. Comfortable in a work environment where urgent issues arise often, and where quickly resolving issues has immediate security value for our end customers
● General knowledge of cybersecurity (or more) and interest in working “upstream” close to the threat actors who are compromising computers/networks
● 2+ years experience in software engineering using Python:
○ You’ve worked in a professional software engineering environment that involves thorough code testing and review processes
○ Strong problem solving capabilities and experience troubleshooting data issues within your own processes and those of your peers.
● Experience with MongoDB, ElasticSearch, and AWS cloud services is highly desirable, as is experience using and working with Telegram