Mayank Sharma

1. What AI capabilities must we measure?

I’m interested in identifying AI capabilities that matter, grounding them in literature from the human sciences, and breaking them down into constructs that are theoretically meaningful and machine measurable.

2. How do we measure them rigorously?

I’m interested in working with annotators, designing interfaces, understanding disagreement, and building human-in-the-loop processes to create benchmarks and evaluation datasets. I apply psychometrics (IRT, validity, reliability, and measurement theory) to evaluate whether our systems capture what they are intended to measure.

3. Do systems built to demonstrate these capabilities have real-world impact?

Once we define what to measure and build rigorous evaluation methods, I draw on research methods from the social sciences to test whether these signals hold in real-world settings, linking evaluations to deployment through causal/experimental methods.

Skills & tools

Programming: Python, R, SQL, Jupyter, Git, LaTeX, Cursor

Data / ML: NumPy, Pandas, Scikit-Learn, PyTorch, Transformers, and NLTK

AI: Benchmark & Evaluation Design; Post-Training (QLoRA, RLHF/DPO); RAG; Prompt Engineering; Human & LLM Annotation

Research Methods: Item Response Theory; Dimensionality Reduction (Factor Analysis/PCA); Reliability & Validity; Regression Analysis; Experimental & Quasi-Experimental Designs; Structural Equation Modelling