40 is still within the normal range, and doctors usually do not worry even about more than 40 as long as it's not high above!

It could also be that you have had some viral infection some time ago and the lymphocytes were high, and now they are going down.

Cancer would definitely show a more extreme result!