Do not try to invite only the famous people or the people doing the "top" work. This usually biases one towards more established researchers (as in, older) and this alas also usually is accompanied by distortion of diversity.
DO try to invite people across the breadth of career stages. Meetings to me should not be only about getting the PIs whose labs are doing the best work to talk. It should also be about giving opportunities to junior researchers - PhD students, post docs and junior faculty who are doing exciting work - perhaps more focused or smaller scale - but nevertheless exciting. If one opens up a invited speaker list to people at diverse career stages one generally greatly increases the gender and ethnic diversity.