Lead/Senior Data Engineer


Job Title
Lead/Senior Data Engineer
Job Description
  • Build and manage the data processing pipeline
  • Ensure reliable processing of data, keeping error counts in check, add data validation checks, and ensure performance of data pipeline is good.
  • Responsible for building the infrastructure for running the data pipeline
  • Work with data ingestion and other feature teams to ensure the data pipeline runs smoothly.
  • Telecommuting is OK
  • No Agencies Please
  • You are currently a technical lead managing a very large scale data processing pipeline.
  • You are a hands-on technical leader
  • You have 12+ years of professional software development experience
  • You have 6+ years of experience with developing data applications in Python.
  • You have a deep knowledge of building, scaling and managing a large scale real-time data pipeline (processing 100s of millions of events per day).
  • You have a deep knowledge and experience working with two or more of the following.
  • Large scale message queues like Kafka / Kinesis
  • Real-time stream processing systems like Spark, Apache Flink, Kinesis Data Streams
  • Have experience in Airflow
  • You have experience with understanding the trade-offs between scalability, reliability, -developer/ops overhead and infrastructure cost of building scalable data pipelines.
  • You have expert knowledge of AWS. Should have built or managed a large scale data pipeline hosted within AWS.
  • You have significant experience working with either SQL based databases (PostGres, etc) or MongoDB.
  • You have the ability to define the roadmap and drive execution of the team.
About the Company

This is a Full Time Opportunity with one of our client, who are into the AI and ML space. They have their own patented products and have got fundings in Millions recently.

Contact Info

👉 Please mention in your application that you found the job on pyremote, this helps us get more companies to post here!

This job is sourced from Python.org Jobs. When clicking on the button to apply above, you will leave pyremote and go to the job application page. pyremote accepts no liability or responsibility as a consequence of any reliance upon information on there (external sites) or here.