Overview

A unique opportunity to join Bing Search, a global search engine powering billions of searches daily, both from humans and from Large-Language Models.

The Bing Metrics team is looking for passionate applied scientists to work on the new generation of metrics and quality control for the Bing Grounding API. The team ensures that Bing returns high-quality, error-free, and authoritative results using a variety of different approaches. Our team builds complex pipeline including crowd judging and machine learning steps to verify our suspicions. Now, we actively use LLMs like ChatGPT as a judge to evaluate the quality of search results at multiple levels: query, answer, whole page and generate insights for the teams who are responsible this experience.

As a part of an international and distributed team you will be responsible for RAG quality metrics within Bing Search. The job provides you with the opportunity to work with multiple teams across entire Bing (>80 different teams) and greatly influence the search engine relevance and search result quality of the entire platform. We are an established core team in Bing with very high visibility and impact.

We are looking for a talented engineer/applied scientist with a passion to work with LLM and specifically RAG, design, implement and test complex data pipelines built on top of LLM models, create new tools for running multi-step prompts to evaluate search engine quality and generate actionable insights for the teams.

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.  

Starting January 26, 2026, Microsoft AI (MAI) employees who live within a 50- mile commute of a designated Microsoft office in the U.S. or 25-mile commute of a non-U.S., country-specific location are expected to work from the office at least four days per week. This expectation is subject to local law and may vary by jurisdiction.

Responsibilities

• Design and implement metrics for RAG with Bing Web Search and other APIs.
• Build pipelines and dashboards for Bing Grounding quality.
• Use LLM models in LLM-as-a-judge settings for data evaluation.
• Engineer prompts for textual and multi-model LLMs for data processing and generation of insights.
• Design and implement E2E pipelines (from sampling anomalies from the logs through prompt engineering to ultimately automatically updatable dashboards).
• Apply classical ML (feature engineering + model training, text and image embeddings) along with LLM to augment data analysis and processing pipelines.
• Help teams to build new innovative search experience with Bing.
  
  

Qualifications

Required Qualifications

• Bachelor's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND hands on experience (e.g., statistics predictive analytics, research)
• OR Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND years related experience (e.g., statistics, predictive analytics, research)
• OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND related experience (e.g., statistics, predictive analytics, research)
• OR equivalent experience.
• Proficient in working with relational and/or non-relational databases, including experience in writing queries and performing data manipulation using SQL or equivalents.
• Experience developing and applying Large Language Models (LLMs) within AI solutions, including Retrieval-Augmented Generation (RAG) techniques for integrating external knowledge sources.
• Ability to work independently, solid collaboration and communication skills.
  

Preferred Qualifications

• Master's Degree in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND hands on experience (e.g., statistics, predictive analytics, research)
• OR Doctorate in Statistics, Econometrics, Computer Science, Electrical or Computer Engineering, or related field AND hands on experience (e.g., statistics, predictive analytics, research)
  
  

#MicrosoftAI #search# #webxt# #LLM# #RAG# #DS#

This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.

Microsoft is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances. If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.