While creating the ASM instance using DBCA in an 11g RAC environment on 64 bit AIX 5L, we got the ORA-27504 error along with others as shown below.
The problem was caused by incorrect UDP and TCP packet settings and is documented in Metalink note 300956.1
As root, the following changes need to be made
# no -o tcp_sendspace=262144
# no -o tcp_recvspace=262144
# no -o udp_sendspace=65536
# no -o udp_recvspace=262144
# no -o rfc1323=1
Adding these entries to the /etc/rc.net will ensure that these parameters take effect on every machine reboot.