The mathematician is among three Massachusetts scholars who were awarded the 'genius' grant — $800,000 over five years.