George gained custody of Peter, who was taken from his biological mother and brought up by his adoptive paternal grandmother, Laura Finch (formerly Black) in Vaucresson, France. In 1925 Laura took Peter with her to Adyar, a theosophical community near Madras, India, for a number of months, and the young boy lived for a time in a Buddhist monastery. Undoubtedly, as a result of his childhood contact with Buddhism, Finch always claimed to be a Buddhist. He is reported to have said: "I think a man dying on a cross is a ghastly symbol for a religion. And I think a man sitting under a bo tree and becoming enlightened is a beautiful one."
He had a small role as an Australian prisoner of war in The Wooden Horse (1950), the third most popular film at the British box office in 1950. His performance as a Pole in Daphne Laureola led to his casting as a Polish soldier in The Miniver Story, the sequel to the wartime morale boosting film Mrs. Miniver; unlike its predecessor, it was poorly received critically.
Finch's next three films saw him support notable female stars: Sophia Loren in Judith (1966), Melina Mercouri in 10:30 P.M. Summer (1966) and Julie Christie in Far from the Madding Crowd (1967). He was reunited with Aldrich for The Legend of Lylah Clare (1968). The Red Tent (1970) was an expensive international adventure film, with Finch as Umberto Nobile.
Finch was married three times. In 1943, he married Romanian-born French ballerina Tamara Tchinarova; they worked together on a number of films. They had a daughter, Anita, born in 1950. They divorced in 1959, after she discovered his affair with actress Vivien Leigh in California. He then married South African-born actress Yolande Turner (née Yolande Eileen Turnbull); they had two children together, Samantha and Charles Peter. During their marriage, Finch had an affair with the singer Shirley Bassey. Bassey had a daughter, also named Samantha, born in 1963; Bassey's husband at the time, the openly gay film producer Kenneth Hume, believed that Finch was her biological father. Finch and Turner divorced in 1965. In 1972 Finch married Mavis "Eletha" Barrett, who was known as Eletha Finch. They had a daughter together, Diana.
During this time, Finch continued to appear on stage in various productions while under contract to Olivier. Finch's closeness to the Olivier family led to an affair with Olivier's beautiful but increasingly unstable wife, Vivien Leigh, which began in 1948, and continued on and off for several years, ultimately falling apart due to her deteriorating mental condition.