The NAND loader seems to support both encrypted and plain-text firmwares. Should work fine to store an unencrypted firmware in the NAND. Maybe true for the USB loader as well. If the firmware starts with the letters "vm" (from vmlinux.bin) the loader assumes it's plain-text. Anything else and it's assumed to be encrypted.
Files loaded over TFTP is always encrypted in the same manner the initrd image in the flash is. There is no tools on the readynas for performing this encryption, but you can find tools for performing these crypto functions from my web page at http://www.henriknordstrom.net/code/readynas/
Alternatively you can bypass the encryption requirements by using a serial console and runtime-patch the bootloader to accept unencrypted files over tftp. See the following page for details:
Note: for iboot 1.00a042 as used in the Duo the rn_nops load address is 0×878326d8
Note: If the code ends by jumping to 0x40000000 then the bootloader conveniently restarts.
_start: call main nop sethi %hi(0x40000000), %g1 jmp %g1 nop