Big Data Developer

PipeCandy is a 'one of its kind', 'data science' driven market intelligence platform that tracks the global eCommerce landscape. Our insights are used by well known global brands and startups. We are venture funded by India, the US, and Singapore based investors.

About the Role:

PipeCandy is building a market/industry intelligence product that combines sophisticated ML/AI algorithms with deep sector intelligence focused on eCommerce and Digital Native companies. We are looking for a super smart Big Data platform engineer to help build our next generation AWS Big Data based platform.

Key Responsibilities:

• Design, construct, test and maintain highly scalable data management systems
• Develop processes for deploying analytical models in production
• Integrate new data management technologies and software engineering tools into existing structures
• Employ custom software (services, microservices, etc.), languages (Python, etc.) and tools to marry systems/components together
• Recommend ways to improve data reliability, efficiency and quality

Skills Required:

• Should be strong in pyspark with respect to rdd functions, dataframes, datasets and sparksql in both streaming and batch
• Ability to use a wide variety of databases (both SQL and NoSQL) and tools
• Experience in implementing spark ETL in AWS using EMR/Glue and related tools
• Exposure to AWS tools like lambda, redshift, kinesis, ec2 and ecs is a plus
• Experience in handling different file types like json, parquet, orc, csv, flat files and databases in pyspark

Qualifications & Competencies Required:

• 3-5 years experience with Big Data tools like spark, elastic search, kafka
• Strong background in building data lake and data warehouse for big data platform

Perks:

• Flat organization structure with an opportunity to work very closely with the founders
• Access to learning, training sessions outside of your immediate line of work
• Access to group kindle account with latest titles
• Stocked pantry, of course