Grease a 9x12-in. baking pan with the olive oil. Place the asparagus, bread, scallions, mushrooms and peppers into the greased pan. Toss the ingredients to evenly distribute.
Combine the milk, eggs, Dijon and Gruyere cheese in a large bowl. Make sure the eggs are completely incorporated into the milk.
Pour the egg mixture over the dry ingredients. Let the casserole set for at least 2 hours in the refrigerator or overnight, so that all the liquid is absorbed.
When ready to bake, preheat the oven to 350°F/180°C.
Remove casserole from the refrigerator and let stand for 20 minutes before baking.
Top with the remaining Gruyere cheese and bake for 1 hour or until a toothpick inserted into the middle of the casserole comes out clean. Serve immediately.