About Mistral
- At Mistral AI, we are a tight-knit, nimble team dedicated to bringing our cutting-edge AI technology to the world.
- Our mission is to make AI ubiquitous and open.
- We are creative, low-ego, team-spirited, and have been passionate about AI for years.
- We hire people that foster in competitive environments, because they find them more fun to work in.
- We hire passionate women and men from all over the world.
- Our teams are distributed between France, UK and USA
Role Summary
- We are seeking a skilled and motivated Web Crawling and Data Indexing Engineer to join our dynamic engineering team.
- The ideal candidate will have a strong background in web scraping, data extraction, and indexing, with a focus on leveraging advanced tools and technologies to gather and process large-scale data from various web sources.
- The role is based in Paris or London
Key Responsibilities
- Develop and maintain web crawlers using Python libraries such as Beautiful Soup to extract data from target websites.
- Utilize headless browsing techniques, such as Chrome DevTools, to automate and optimize data collection processes.
- Collaborate with cross-functional teams to identify, scrape, and integrate data from APIs to support business objectives.
- Create and implement efficient parsing patterns using regular expressions, XPaths, and CSS selectors to ensure accurate data extraction.
- Design and manage distributed job queues using technologies such as Redis, Kubernetes, and Postgres to handle large-scale data processing tasks.
- Develop strategies to monitor and ensure data quality, accuracy, and integrity throughout the crawling and indexing process.
- Continuously improve and optimize existing web crawling infrastructure to maximize efficiency and adapt to new challenges.
Qualifications & profile
- Bachelor’s or master’s degree in computer science, information systems, or information technology
- Strong understanding of web technologies, data structures, and algorithms.
- They should have knowledge of database management systems and data warehousing.
- Programming Languages: Proficiency in programming languages such as Python, Java, or C++ is essential.
- Masterings of Web Technologies: Understanding of HTML, CSS, and JavaScript is crucial to navigate and scrape data from websites.
- Knowledge of HTTP and HTTPS protocols
- A good understanding of data structures (like queues, stacks, and hash maps) and algorithms is necessary
- Knowledge of databases (SQL or NoSQL) is important to store and manage the crawled data.
- Understanding distributed systems and technologies like Hadoop or Spark Experience using web Scraping Libraries and Frameworks like Scrapy, BeautifulSoup, Selenium, or MechanicalSoup
- Understanding how search engines work and how to optimize web crawling.
- Experience in Machine Learning to improve the efficiency and accuracy of web crawling
- Familiar with tools such as Pandas, NumPy, and Matplotlib to analyze and visualize data.
Benefits
- Daily lunch vouchers
- Contribution to a Gympass subscription
- Monthly contribution to a mobility pass
- Full health insurance for you and your family
- Generous parental leave policy