Be a natural - Let things happen naturally. As you read more, and convert the read word into spoken word, your speed will gradually pick up. Do not be artificial.
Read, read, read - As you read more, your confidence about issues / topics / subjects will grow. You will start owning knowledge. You will develop an inner, natural strength. It is a matter of time that is starts showing on your face.
Write at least 1 essay per day - It is the ultimate test of what you are learning, and if you are learning something at all. Pick a plain paper, think of a topic, and write an essay (300 words max) on it. Do it every day. It will very productively force you to raise the level of the game.
Be measured while you speak - Do not try to outgun anyone in the group. Take a few seconds before you react (2 or 3 seconds, say). That will ensure that you do not over-react, instant-react, or wrong-react.
Talk to yourself - You will never be ashamed doing that. Keep doing it regularly. It will open up your tongue.
Shout, at least once a day - Go to the rooftop literally, and shout aloud. Do that for some minutes. Speak something sensible if you can. It will train the mind not to be scared of anything. Of course, if you do that elsewhere, people may beat you up.
Love yourself for what you are - This is perhaps the most crucial step. If you hate yourself for not being a fluent English speaker (as per whatever random benchmark you've adopted), your sense of self-worth will be destroyed. Don't let that happen to you.
I have come across some really fast speakers who talk sense, and some who are idiots; some measured speakers who talk a lot of sense; and some slow speakers who are a pleasure to listen to.
It all depends on your profession, your eye-contact and body language, and the choice of words + depth of knowledge.
So relax, and start your personal journey of success! It will take time, so buckle up. Wish you great luck.