I am in agreement with you that it's "my hardware", but if you read their agreement (mostly aiming at XBox Live I assume) one could assume that modifying hardware might result in getting banned from XBox Live.
I guess that's the big deal for most users.
Granted, replacing a hard drive with a larger model should not be a problem in my opinion, but ... Microsoft could take it as an excuse in the future to block you, or not.
See: [http://Software License Terms|https://support.xbox.com/en-US/xbox-one/console/software-license-terms]
1. (c) “'''Authorized Accessory'''” means a Microsoft branded Xbox One, Xbox 360 or Xbox hardware accessory, and/or a Microsoft licensed, third-party branded, Xbox One, Xbox 360 or Xbox hardware accessory whose packaging bears the official “Licensed for Xbox” and/or the “Designed for Xbox” logo.
2. (i) You will not use '''Unauthorized Accessories''' or Unauthorized Games. They may not work or may stop working permanently after an Xbox Software update.
2. (iii) You will not attempt to defeat or '''circumvent''' any Xbox Console, Kinect Sensor or Authorized Accessory '''technical limitation''', security, or anti-piracy system. If You do, Your Xbox Console, Kinect Sensor or Authorized Accessory may stop working permanently at that time or after a later Xbox Software update.