The most basic element of exercise is the combination of stress and recovery. This is the "fundamental theorem of fitness" if there ever was one. When the body is pushed, it is broken down. Afterwards, it adapts, and then you are stronger the next time you try that same thing. They go together, work and rest, and you really can't have one without the other.
If all you do is rest, then obviously you wont improve. But all work is also self-defeating; it is during these times of break that our body actually rebuilds itself, it is during these times that you grow. If you work yourself ragged every day and never let up, you are constantly breaking the machine down, but never allowing it a chance to repair. It's a shame, because you are skipping the easiest part. If you worked hard, you earned your rest. So take the time and enjoy it! You may not have thought that lying on the couch watching TV could be part of a fitness program, but it could be every bit as necessary as the long run.
But when do you push yourself, and when do you take it easy? Let's answer the last question first. You need to rest when you feel tired, sore, worn-out, or when something hurts. The amount of time will depend on what shape you are in and what your recent workouts were like. For a beginner, this may be for several days. I'm in pretty good shape myself, but last week I did a strength training workout that I hadn't done in a while, and I was majorly sore for 4 days following. After a day or two I wanted to get out on the road again, but I knew that the soreness was a sign that my body was still rebuilding and I needed to keep resting. So listen to your body and rest as long as you need until you feel "normal" again. Now the answer to the first question becomes clear: you push yourself any other time.
Many other decisions come down to this distinction between stress and recovery, and the more you listen to your body, the better able you will be to know what you need.