It is in part an empirical issue. Join any Ron Paul meet-up group on line and you can see who is making posters and flyers and holding rallies and what they say. In my area they are not racists. They are all independents who are small "l" libertarians. Libertarians should be networking them