Through a little research here are some cool science facts.
Volumetrically speaking, Aluminum is more affected by temperature than brass is. That is, Aluminum will show more shrinkage per drop in degree of temperature.
However Aluminum is a better conductor of heat so it will actually warm up quicker than brass.
Not sure about composites or wood as their material makeups are a bit more complicated.
Theoretically if you had two whistles of exact same dimensions and design with one being aluminum and one being brass, the aluminum one should always be a bit flatter in cold weather. However it will take less time for it to literally "warm up".
edit: I just forgot that woods are great insulators. And shrinkage may vary greatly, so woods are probably always best played when you
know they are warm.
Delrin is even less affected by thermal expansion/contraction than brass or aluminum. However it does take the longest to "warm up".
So if you find yourself running late to a session and you are carrying your whistles in out of the cold, start with a composite as it will play slightly better on average than any metal whistle. But if you have a few minutes to say hello and order a beer then pull out your aluminum once you sit down to play