The ideas that inspired the themes and details depicted in the James Bond books occasionally become apparent after researching material such as Ian Fleming's Sunday Times articles, the books that he placed in Bond's library, or copies of interviews with Fleming. Sometimes, however, Fleming is pretty clear himself on the background to his books.
The Lilly Library of Indiana University holds all the original manuscripts of the James Bond novels, except Thunderball, The Man with the Golden Gun, and Octopussy and the Living Daylights. The manuscripts include Fleming's copious annotations, added not only during his usual period of revision in the months after completing his first drafts, but sometimes years after publication. These provide much information on Fleming's sources and inspirations.
On the manuscript for Casino Royale, Fleming wrote that some of the incidents were based in fact. These included the botched attempt on Bond's life using the trick camera, which was based on similar events committed by Russian agents in Ankara, Turkey. Fleming noted on the manuscript of Live and Let Die that the settings that featured and the details about fishing were based on his own experiences. He had also consulted the New York Police Department and the US Navy Department for information.
Fleming drew much of the background to the Dover scenes in Moonraker from personal experience too, as he had a home at nearby St Margaret's Bay. Fleming wrote on the draft of From Russia, with Love that he drew on information from the Soviet refugee and spy, Tokaev, for the Russian background, and claimed that fights between gipsy women can be arranged for a small fee.
Much of the background described in You Only Live Twice was based on personal experience, but he was also helped by Richard Hughes, the Sunday Times far eastern representative, and the editor of the annual, This is Japan. Fleming also referred to four books: Meeting with Japan and The Diving Girls' Island, both by Fosco Maraini, The Heart of Japan by Alexander Campbell, and The Horned Islands by James Kirkup.
Short of visiting the Lilly Library and examining the manuscripts, a visit to the library's website is the next best thing. If you haven't already seen its online catalogue of the Fleming collection, I urge you to do so, as it provides much insight into the background to the Bond books.