Achieved a 99.9% increase in efficiency by reducing time-to-delivery for new data pipelines from thousands of hours to minutes.
Developed a Python script that streamlined pipeline generation and metadata cleaning for over 1300 individual tables across various ingestion and consumption pipelines, collaborating with the financial risk management team.
Crafted SQL queries in Snowflake to create precise lists of table names, ensuring seamless data transfer, accuracy, and integrity.
Worked with machine learning engineers to develop pipelines that helped streamline data
Technologies
Designed, developed, and deployed scalable data pipelines in a CI/CD environment using a comprehensive tech stack:
