The charging effect due to the fluctuations of surface potential (Vs) can cause fluctuating granularity (bee-swarm effect), e.g. sometimes seen in the TEM image of a thin insulating film .
The drift and jump of EEL spectra or images can be induced by different reasons:
i) Induced by charging as a result of the exposure to the electron beam:
i.a) The EEL spectrum or image first gradually drifts in one direction when the charge is building up;
i.b) It suddenly jumps in the opposite direction when the charge dissipates with a discharge.
If something is really charging, then increasing the beam current while keeping the illuminated area on the viewing screen the same should lead to an increased rate of drift.
A small selected area diffraction (SAD) aperture can be used in order to determine whether the charging occurs in the camera chamber or in the GIF. The small SAD aperture ensures that all the electrons enter the GIF and none can hit and charge any parts in the camera chamber. If the charging continues, then it must be in the GIF. Otherwise, it should be in the camera chamber if it stops.
ii) Induced by unstable high voltage of the electron beam.
The charging effect in TEM specimens can be reduced by lowering the electron beam intensity. On the other hand, charging effect is more significant for small beam sizes due to generation of secondary electrons.
 Curtis, G.H., Ferrier, R.P., 1969. The electric charging of electronmicroscope
images. J. Phys. D 2, 1035–1040.