A good time to harvest peppers is when they are full size and have matured to their full color. Well that's easy to say, but another thing when looking at the plant full of fruit. The best thing to do is to cut one pepper off the plant and taste it. If it is bitter and tastes a little green, wait another week and try again. A general rule is to use the number of days to maturity, but that will vary in different growing conditions. Both sweet and hot peppers can be eaten at all stages of growth but are most flavorful when mature and fully ripe.