Supervisor threatening to take me(a preloader) off of the local sort shift

I've been doing preload for about 6 months and recently I've been taking up doing local sort after my morning shift and now lately my supervisor has been telling me my performance is declining which I believe it isn't its been the same I've shown up everyday I do my job I may stack out a bit but I load it nonetheless can he in fact take me off of another shift? Or can I file a grievance?? Edit: He's the building manager not the supervisor!!! My bad