In 1953 he appeared in his first Hollywood movie, Elephant Walk (1954), shot in Ceylon and Los Angeles. The circumstances of production were turbulent; filming started with Vivien Leigh in the lead but she had a nervous breakdown during production and had to be replaced by Elizabeth Taylor. The experience helped sour Finch on a Hollywood career and he would only work occasionally there for the rest of his career.
Finch was married three times. In 1943, he married Romanian-born French ballerina Tamara Tchinarova; they worked together on a number of films. They had a daughter, Anita, born in 1950. They divorced in 1959, after she discovered his affair with actress Vivien Leigh in California. He then married South African-born actress Yolande Turner (née Yolande Eileen Turnbull); they had two children together, Samantha and Charles Peter. During their marriage, Finch had an affair with the singer Shirley Bassey. Bassey had a daughter, also named Samantha, born in 1963; Bassey's husband at the time, the openly gay film producer Kenneth Hume, believed that Finch was her biological father. Finch and Turner divorced in 1965. In 1972 Finch married Mavis "Eletha" Barrett, who was known as Eletha Finch. They had a daughter together, Diana.
He did radio acting work with Hugh Denison's BSA Players (for Broadcasting Service Association, later to become Macquarie Players). He came to the attention of Australian Broadcasting Commission radio drama producer Lawrence H. Cecil, who was to act as his coach and mentor throughout 1939 and 1940. He was "Chris" in the Children's Session and the first Muddle-Headed Wombat.
During this time, Finch continued to appear on stage in various productions while under contract to Olivier. Finch's closeness to the Olivier family led to an affair with Olivier's beautiful but increasingly unstable wife, Vivien Leigh, which began in 1948, and continued on and off for several years, ultimately falling apart due to her deteriorating mental condition.