Maximum drive capacity in optical bay?

The limit here is the drives height. Some carriers can support a 12mm drive most can only support a 9.5mm. As we don't know the given holder or drive details (can you give us what you are getting) I can't offer you more.

You may want to think this though again. One of the issues you will face is the drive's SATA I/O speed. Remember the optical drives port is IDE not SATA so what ever drive you put into it it will be slow and you may in fact have an issue if you put in a SATA III (6.0 Gb/s) HD as many of these converting holders SATA port is only SATA I (1.5 Gb/s) or SATA II (3.0 Gb/s) so the drive may fight the SATA port as well.

If you want speed how about swapping out your HD for a SSHD (hybrid drive) Then you get the zip of the deep SSD cache with the storage of the HD. If you need more space they even have 3 & 4TB SSHD 3.5" drives! Checkout Seagate's drives.