Generally stated, taking the mean of many measurements is the way towards higher confidence in the result. And if the measurements differ in their own reliability, a weighted mean is in order.
It is a common technique to mean several GPS observations over time, at a static location, to gain higher accuracy. In your case, however, you are moving so it is no longer a simple case of averaging over a (relatively long) time interval.
Depending on how fast you are moving and how frequent your observations are being made, what you could do is compute a weighted, moving average of the latest N observations. Suppose N is four (you may need a very different number depending on the speed of movement and frequency of observation), then
WMA P(T) = 10% P(T-3) + 20% P(T-2) + 30% P(T-1) + 40% P(T)
where WMA is weighted moving average
and P(T) is position at time T
There are much more statistically rigorous methods, Kalman filtering and smoothing, for example, but the above should give you a basic idea on how to weight your observations -- lat and long -- as you progress in time, giving higher weight to the most recent ones.
One simple improvement, if you can accept post processing, is to also use future observations. In that case, at any given time, you use the N previous observations and the N next observations, again, weighting the more recent ones more heavily. (Just make sure your weights add up properly.)