Inaccuracies thus introduced may be a real problem and your only option would be to use a camera that you can pair with a GPS or one that has a built-in GPS. In that case the location data embedded as metadata will no longer depend on matching GPX tracks with your camera clock.
Personally I have been never used GPX tracks, matching them with EXIF time stamps. Therefore, I have no idea how accurate this actually works. I am, however, regularly using a GPS paired with my camera via bluetooth and have always been surprised how accurate this is.
Historically, I believe it is true that the US military scrambled their GPS signal, but today that no longer seems common practice. The US government website (http://www.gps.gov/systems/gps/performance/accuracy/) states: "...For example, the GPS signal in space will provide a "worst case" pseudorange accuracy of 7.8 meters at a 95% confidence level. [...] Real-world data from the FAA show their high-quality GPS SPS receivers attaining better than 2.168 meter horizontal accuracy, 95% of the time."
That is pretty darn accurate, meaning: if you have somewhat decent GPS receiver it will typically show your location pretty much where you are - only being a few meters off...
About military vs. civilian use their website states: "The accuracy of the GPS signal in space is actually the same for both the civilian GPS service (SPS) and the military GPS service (PPS). However, SPS broadcasts on one frequency, while PPS uses two. This means military users can perform ionospheric correction, a technique that reduces radio degradation caused by the Earth's atmosphere. With less degradation, PPS provides better accuracy than the basic SPS. "
And in the "real world"?
In my own experience even in remote places like the Galapagos finding the same sites marked by GPS these days works astonishingly well! If you are in the business of building roads you probably need more accuracy, but tracking back my own course using my GPS typically deviates only a few meters from my original path.
That of course varies by the GPS receiver you'd be using and, presumably, its built-in chip. In my experience for example, funnily enough, my Bad Elf Pro, a small GPS paired via foolography bluetooth adapter with my camera actually appears to be more accurate than the comparatively clumsy and large GARMIN GPSMap 62s that I am regularly using as well. In any case, both are typically fairly accurate, only "off" within a few meters.
In places where WAAS ground stations signals can be received that accuracy can supposedly be even be improved further. Near the equator this reportedly does not work very well. Again in practice I have often found these discrepancies largely academic. The question is: how much accuracy do you really need?
If you take a photo and then use the embedded GPS data to track back to the locality where that photo was shot you quite likely will recognize that locality again - even if you are not standing in the very exact same spot. And if you actually need "geodesic" accuracy for your photo coordinates, best get yourself a high accuracy GPS and manually copy-paste the coordinates into the photos
[Regarding the recent launch of Galileo by the Europeans: embarrassing. Not only has that system been delayed many years, even now, after its official launch date barely enough satellites are up there to provide any decent signal. And then: virtually no receiver that can actually catch these signals are today even being sold. Yes, the system is supposed to be technically far superior to both GPS and Glonass, fair enough. But what use is the technically most advanced system if it takes forever to get implemented ...]