Have you noticed whether those FPS issues were happening on official maps or third-party maps?
A lot, in fact most of third-party maps that are hosted on popular servers are very poorly optimized for the Source engine, resulting in low framerates even on the beefiest computers out there. Your rig is hardly the problem in this case, something else is going on.
If your frame drops are happening on official Goldeneye Source maps, then something unusual is definitely happening, but knowing on which map it happens will help narrow the problem. If we would couple that fact with it happening only on select servers, then perhaps the architecture of those servers are having problems and are in need of a restart. Have you tried playing on a local server with bots or other servers to see if the framerate issues persist?