Yes, since much of the Crown's content involves unrecorded discourse within personal relationships, those elements alone are pure speculation. In effect filling in the spaces between the dots. Much of the remaining content is almost certainly opinion, recollections or speculation gleaned from various sources, some good some less so. Then there's the artistic licence element added make the whole thing more dramatic. What's left are the bare facts which form the back bone of the whole Netflix Crown show.
The Royals certainly won't fill in the gaps so Netflix have done their version. Nothing wrong with that as long as it isn't treated as an accurate portrayal of events therein.
Accepting it for what it is I must say that it is an extremely well made sumptuous production and one that we have both thoroughly enjoyed.