From “Popeye” Doyle to Royal Tenenbaum, the actor's five-decade career showcased his ability to portray complex characters with emotional depth.