Developer – Web Crawling and Data Acquisition

Location: Chennai

Experience: 1-3 Years

PipeCandy is a 'one of its kind', 'data science' driven market intelligence platform that tracks the global eCommerce landscape. Our insights are used by well known global brands and startups. We are venture funded by India, the US, and Singapore based investors.

About the Role:

We are building a complex data product that aims to revolutionize industry intelligence by applying sophisticated ML & AI algorithms on millions of data points.

We are looking for a crawler who will work on large scale web crawling and custom data acquisition. If you love playing with data and code, have an eye for detail and a strong client delivery mindset, we’d love to hear from you.

This position requires experience in technology related to web crawling and data scraping, the ability to understand business requirements, develop data acquisition code as per specification, and managing project delivery.

Key Responsibilities:

  • Design, develop, and configure web crawling software systems to support our crawling and data extraction requirements
  • Implement web scraper and data extraction for various websites as per business requiements
  • Monitor, test and maintain web scraper code to address any changes in crawled website
  • Implement best practices and patterns in design, development, deployment and software testing

Skills Required:

  • Web Crawling and data scraping experience
  • Knowledge of scraping frameworks such as Python (Request, BeautifulSoup, etc), Web- Harvest and others
  • Good to have experience in JavaScript, JQuery, PHP & Ruby on Rails
  • Strong coding experience in Python (Knowledge of Java, Javascript is a plus)
  • Experience with SQL and NoSQL databases
  • Experience with cloud based crawling platforms
  • Experience with multi- processing, multi- threading and AWS/ Azure

Competencies Required:

  • Qualification – Graduate, preferably B.E. or MCA background
  • Hands-on experience in coding & development in related technologies
  • 1- 3 years of experience


  • Ability to work with business and technology teams to deliver data acquisition projects
  • Ability to multi-task, solve problems and think strategically
  • Strong communication and collaboration skills


  • Flat organization structure with an opportunity to work very closely with the founderss
  • Access to learning, training sessions outside of your immediate line of work
  • Access to group kindle account with latest titles
  • Stocked pantry, of course