Caste of Vashista - the saptarishi

The Saptarishi (from saptarṣi, a Sanskrit dvigu meaning "seven sages") are the seven rishis who are extolled at many places in the Vedas and Hindu literature. The Vedic Samhitas never actually enumerate these rishis by name, though later Vedic texts such as the Brahmanas and Upanisads do so.

The list varies among the various brahmanas and upanishads as given below

1. The Shatapatha Brahmana and Brihadaranyaka Upanishad (2.2.4) acknowledge the names of seven rishis (or Saptarshis) as: Atri, Bharadvaja, Gautama, Jamadagni, Kashyapa, Vashista, Vishvamitra
2. Krishna Yajurveda in the Sandhya-Vandana Mantras has it as: Angiras, Atri, Bhrigu,  Gautama, Kashyapa, Kutsa, Vashista
3. Mahabharata gives the Seven Rishis’ names: Marichi, Atri, Pulaha, Pulastya, Kratu, Vashista,  Kashyapa
4. Brihat Samhita gives the Seven Rishis’ names as: Marichi, Vashista, Angiras, Atri, Pulastya, Pulaha, Kratu

 Now few of these rishis are mind born sons of brahma. Brahma's mind born sons are the seers Marici, Atri, Angiras, Pulastya, Pulaha, Kratuj, Pracetas, Vashishta, Bhrgu and Narada. From Brahma's body came his nine sons Daksa, Dharma, Kama, Anger, Greed, Delusion (Maya), Lust, Joy, Death and Bharata and one daughter called Angaja.

The interesting thing to note is Vasistha figures in all these list is so by traditions of brahmanas and upanishads vashista one of the sapatrshi is mind born son of brahma.

Rig Veda, Tenth chapter, purusha sukta clearly says that brahmana is born from brahma's mouth, kshatriya from arms, vaishya from body and sudra from feet.

So Vashista is not a brahmana or kshatriya or vaisya or sudra.

So any descendants of Vashista will not fall into any of these four varnas, that includes Vyasa and his descendants pandavas and kauravas