Does it matter if it is, would it make any difference at all if everything else around your conscious mind is, say, a computer-based 3D animation? I think not, in ordinary life it wouldn't make much difference.
That said I wondered about this too when I was about 10-11 years old, and have since realized that lots of people have thoughts like that at one point or another. Like Stephen Hawkings probably; I heard he hosted a time-travellers party but nobody came (the invitation to the party was sent out after it had happened).