Thank you Venerable @Bensi for your valuable sharing!
On your question, I do think Buddhists belong to an “elite group”.
Elitism has a somewhat negative connotation in society but in this case you qualified it by what this group can do - the potential to help all living beings.
I would add that this group is “elite” also because of the small number of people that has the fortune to meet Teachers that can led them on this path.
I am glad to have encountered these Teachers.