Remove stems from mushrooms.
Coarsely chop stems and set aside.
Cook bacon in heavy large skillet over medium heat until crisp.
Using tongs, transfer bacon to paper towels and drain.
Pour off all but 1 tablespoon drippings from skillet.
Add chopped mushroom stems, garlic and rosemary and sauté over medium heat until tender, about 10 minutes.
Crumble bacon and add to skillet.
Add olives and stir to combine.
Mix cream cheese in medium bowl until smooth.
Add mushroom mixture and stir to blend.
Season to taste with salt and pepper.
Lightly brush rounded side of mushroom caps with oil.
Place rounded side down on large baking sheet.
Spoon filling into caps, mounding in center.
(Can be prepared 1 day ahead. Cover stuffed mushrooms with plastic and chill.)
Preheat oven to 375° F.
Bake stuffed mushrooms until heated through, about 20 minutes.
Transfer mushrooms to platter and serve.