Fantastic primitive campground near the entrance to Anza Borrego. Wide open spaces with plenty of spots to set up camp, beautiful views, drastic sunrises and sunsets. The icing on the cake -- the inky night sky. There is NO better place in the area to view the stars than Blair Valley. The "campground" has pit toilets which are usually clean as well which is great for those who aren't ready to squat. There are many nice dirt roads and trails to explore. Check out the Pictographs trail nearby.
If you're coming from San Diego/LA, you drive past Julian. That's a great place to stock up on supplies and get gas. There's also a few stops along the way in Agua Caliente. Bonus -- there's decent cell reception in the area.
Lastly, getting here is pretty easy. I've done it in a slammed RWD sedan and made it through fine. Now I've got a 4x4 and of course, that's what you *should* use.