I'm doing it. I, too, don't mind little bits of soy, but I certainly don't want to base my diet on soy. I guess I can't say I'm vegan, yet. *mostly* vegan might be a better description.
My meals are based on vegetables, grains (usually rice, quinoa, or bread), and legumes (beans of all sorts, peanuts). I have some nuts and seeds occasionally, as well. Soy is really only a part of the meat/dairy replacement products, and if you don't eat those there's no worry. But even if you want replacements, there are nut based milks and cheeses, wheat meat, etc. You can pretty easily avoid obvious soy if you don't mind it as a small ingredient (like soy lecithin as the last ingredient, yk?).
HTH!