I was expecting Outlook to use the regional settings in Windows, but it looks like the regional settings only has a first day of the week option.
Outlook has it's own "first week of the year" setting which defaults to "first week with 4 days". Possibly based on other regional settings such as my country?