I like the idea and don't mind the added complexity, but I do have some balance concerns:
At 20th-level with 133 points a character could have 19 5th-level slots, which seems really powerful.
As early as at 9th-level, a character could have 8 5th-level slots (with a point left over).
Even with Eldritch Invocations, the two 5th-level slots a warlock gets at the same level seem very underpowered by comparison.
So, is anyone using spell points? What issues (if any) have you come across?