Hi and welcome to PureOC... OC's is fairly simple as there are a couple of ways to doing it.
The easiest way is to make the adjustment and settings in your bios.
Make sure to turn off all for power saving features and turbo, so C6, C1E, Turbos and Etc.
Once you are done, you can first try to set your voltages, When I had my 2500k OC, I used 1.40 volts first to test how high I could get with that.
You can then use the Mutiplier. ex: Stock freq should be at 100mhz FSB - x 40 will give you 4.0ghz. What I would do first is try to start at like 4.0-4.4 with 1.4 volts and run Prime95 or Linx to test for stability. If it passes push it up a little more until you BSOD, or it be comes unstable, from there you will know what your limits are with those voltages. That is the easiest way to start.
With your cooling you shouldn't have any problem with those freq.
Once you get more familiar with Ocing, then you can play around with other settings, but first leave everything else on auto and only play with voltages and mutiplier, this way it easy to do trial and error with the least amount if settings being changed.
My highest with my 2500k was 4.8ghz at 1.48volts (using custom water cooling) and CPU NB settings at 1.3 volts. These were stable running prime for 8hrs. Temps were under 60c, I couldn't get my chip past 5.0ghz stable, I was able to boot, but can't do anything else.
Hope some of that helps..
OC your GPU is easy, esp with AMD, you can use AMD overdrive to see what the highest setting you can get it, otherwise in the past I have used a program from Sapphire called Trixx as well. But before you jump into that arena, first get your CPU OC done and stable.