Job Role:
We are looking for an experienced Data Engineer with expertise in Python, Snowflake, and AWS to develop, optimize, and maintain data pipelines. The ideal candidate will work as an individual contributor while collaborating with a global team to build scalable data solutions, APIs, and frameworks for efficient data processing and analytics.
Key Responsibilities:
- Design, develop, and optimize data pipelines using Python, SQL, and JavaScript within Snowflake and AWS environments.
- Build and maintain frameworks using Python and Stored Procedures to simplify design patterns and improve efficiency.
- Develop and maintain Data APIs (Web/REST APIs) to facilitate seamless data integration.
- Collaborate with the DevOps team to standardize the deployment of Python applications.
- Act as the Level 2 support contact for overnight batch issues, resolving escalations from the L1 team.
- Contribute to the migration of legacy systems (Informatica/.NET) to the target tech stack.
- Improve orchestration workflows for better maintainability and performance.
- Maintain and enhance documentation, runbooks, and monitoring frameworks to ensure data completeness.
- Conduct code reviews, impact analysis, and testing for technical changes.
- Collaborate with onsite developers on minor enhancements and troubleshooting.
Required Skills & Expertise:
- Strong experience in Data Warehousing concepts – OLTP, OLAP, Dimension & Fact loading.
- Expertise in Python for data applications, including reading, transforming, and writing datasets.
- Proficiency in Python libraries such as boto3, pandas, numpy, pyarrow, Requests, FastAPI, Asyncio, Aiohttp, PyTest, and OAuth 2.0.
- Experience with multithreading, multiprocessing, and Snowflake Python Connector (Snowpark).
- Hands-on experience in building data APIs (Web/REST APIs).
- Strong experience with AWS services – S3, Athena, Glue, Lambda, SNS, SQS.
- Knowledge of application packaging and distribution using Wheel packages, Zipapp, PyInstaller, and Docker containerization.
- Expertise in Snowflake – SQL, Pipes, Streams, Tasks, Time Travel, Data Sharing, and Query Optimization.
- Experience in Snowflake scripting, including SQL Stored Procedures, JavaScript Stored Procedures, and Python UDFs.
- Understanding of Snowflake internals and integration with reporting & UI applications.
- Experience working in financial services, preferably in buy-side firms.