As the saying goes, there are two kinds of people - in this case, they are separate in the ways they look at the past. The first kind regards it with respect and affection, but they keep their distance from it, too. The past is valuable to them for what it can give them, the lessons and ideas and the whole treasure trove of curious and wondrous strange things 'they' did 'then'. The past here is connected to the present in a million ways, very much like a piece of fabric.
The other kind is much more sentimental and even nostalgic, to the point where the past is far more alive to them than the present and the idea of weaving the past into the future is nearly unimaginable. They may even try to do the impossible - to bring the past back and have things 'just as they were', as much as possible.
This song sits awkwardly between these two kinds of people; its very nonchalance - the first thing you hear is whistling, after all - points to a kind of casual 'keep calm and carry on' attitude that either betrays a rather cold-hearted figure who is unable or unwilling to commit (who writes letters in the sand? The very medium, if I can call it that, suggests impermanence, which goes against the "vow" made on the beach. I know very well that (young) lovers can and do find significance in the most ephemeral and trivial of things, but it's not like writing implements and paper didn't exist in 1957).
There is also the chance, of course, that Boone is in a kind of mindset wherein a broken vow doesn't really mean that much in the greater scheme of things - not so much a Buddhist acceptance of loss as more a Christian conviction that this girl obviously wasn't righteous enough for him, so there is no love lost. Boone sings so lightly and politely that it is hard to hear any heartbreak at all in the song; he sings like a wooden doorstop, all whistling near-indifference, the band behind him fruitlessly trying to inject some notion of what sensual walks by the surf have been lost. But Boone remains by the ocean, seemingly in a kind of limbo, somehow both nostalgic and cut off at the same time. (For those of you wondering, yes there is more Pat Boone to come, but not for a little while.)
(It is worth noting that this was a big transatlantic hit song; it is also worth noting that the two songs which leapfrogged it to number one were all about sexual urgency and intense hormonal longing - "All Shook Up" by Elvis and "Diana" by Paul Anka. Once again, the number two song is utterly in opposition to what is at the top.)