We came in mid May, when all the sites were first-come first-served, and only two other sites were occupied. It was totally silent!
Frazier Park is about 15-20 minutes away, so if you want some low effort camping this is about as convenient as it gets. The road up from the ranger station is a little rough, be careful not to get a flat tire.
Site 23 is nice and flat for tent campers.