Why did Persona drop the Shin Megami Tensei prefix in the West?
The first fact corrects the question's usual framing. In Japan, Persona games were not titled Shin Megami Tensei: the first game carried its own banner, Megami Ibunroku, and Persona 2 dropped even that. The Shin Megami Tensei prefix is a Western addition. After the successful Western release of Shin Megami Tensei III: Nocturne, Atlus's overseas branches added the Shin Megami Tensei moniker to future Megami Tensei releases to help market the games, keeping the whole franchise under a single recognizable banner in the West. Before that decision, the West had tried a different banner first: Revelations, chosen for the first Persona because the team thought it sounded cool, and retired after the Persona 2 era.
So Persona 3 and Persona 4 reached Western shelves as Shin Megami Tensei: Persona 3 and Shin Megami Tensei: Persona 4, not because Tokyo named them that, but because the Western publisher judged that the Nocturne name would help sell them. The man who later explained the decision is on the record by name, in the section below.
Two Western boxes: with the banner, and without it
Japan never made this change, because Japan never had the prefix to drop: Japanese Persona boxes carried no Shin Megami Tensei banner in the first place.
THE DECISION, NAMED
Masaru Nanba, on the recordThe documented explanation comes from Masaru Nanba, a senior project manager at Atlus, in comments preserved in the Persona 4 record. Nanba said it was decided that Shin Megami Tensei was to be kept in the title of Persona 3 and Persona 4, because it was believed that they were part of the same series as Shin Megami Tensei III: Nocturne. The banner was a statement of family, made deliberately, at the moment Nocturne had just taught the West what the family name meant.
THE DROP
Golden and Arena, 2012Nanba's account gives the reason for the removal in the same breath, and it is practical, not philosophical: the Shin Megami Tensei title was omitted from both Persona 4 Golden and Persona 4 Arena, as it would have been much too long. The full Japanese titles show the problem. Persona 4: The Ultimate in Mayonaka Arena and Persona 4: The Golden were themselves shortened for the West, to Persona 4 Arena and Persona 4 Golden; stacking Shin Megami Tensei on top of either would have produced a title nobody could say. The series record confirms the consequence: the shortened, banner free form has been applied to every game in the series since. Persona 5's English release followed the new house style, its record noting plainly that in English speaking regions the Shin Megami Tensei moniker was dropped from the title.
WHAT DID NOT CHANGE
The family, minus the labelNothing about the games changed with the banner. Persona remained a spin-off of the Megami Tensei franchise, sharing its demons, its fusion systems, and its Velvet Room lineage, and it went on to outsell its parent series. What changed was the Western publisher's judgment about what the box needed to say: in 2007 the Nocturne name was the strongest signal Atlus had in the West, and by 2012 the word Persona was. The prefix did its job, which is why it could be retired.