If you allow the tea to steep for over 10 minutes, it will not only ooze out more of the healthy oils and nutrients but also make the tea stronger and give it a bitter taste. If you use a fresh sprig of rosemary tea, you won't need to filter it as you will automatically leave the rosemary needles on the stem before adding it to the boiling water to brew it. On the contrary, if you use dried rosemary, ensure that you don't use more than a teaspoon of the tea leaves for every cup of boiling water.