Although the puzzle seems to refer to some characteristic of people (both bad and good), the lines about 'center of gravity' and 'at ease in depravity' don't really fit this interpretation. Like so many of the classic riddles, this one ties all the lines together by some common element of the words, not their meanings. The solution to the riddle is the letter "V".

With thieves I consort,
With the vilest, in short,
   I'm quite at my ease in depravity;
Yet all divines use me,
And savants can't lose me,
   For I am the center of gravity.