Preheat your oven to 350°F (175°C). Grease a 9x13-inch baking dish with butter or nonstick spray.
Place the brioche bread cubes in the prepared baking dish, distributing them evenly. If you're using raisins, sprinkle them over the bread. Set aside.
In a large bowl, whisk together the whole milk, heavy cream, granulated sugar, light brown sugar, eggs, vanilla extract, cinnamon, nutmeg, and salt until well combined.
Pour the milk mixture evenly over the bread cubes, making sure all the bread is soaked. If using nuts, sprinkle them on top. Let the bread soak for about 15 minutes, occasionally pressing down the bread to ensure it absorbs the liquid.
Bake the bread pudding for 45-55 minutes, or until the top is golden brown and a toothpick inserted in the center comes out clean. Allow the bread pudding to cool slightly while you prepare the sauce.
To make the sauce, melt the butter in a small saucepan over medium heat. Add the brown sugar and stir until dissolved. Slowly whisk in the heavy cream and cook for 2-3 minutes, until the mixture thickens slightly. Remove from heat and stir in the vanilla extract and salt.
To serve, cut the bread pudding into squares and drizzle with the warm sauce.