So I don't know if this is just some odd coincidence or what, but it's worth sharing either way I think. I normally refresh the boards by clicking on the name of the board and not the refresh button. It seems like every other time I click on the board name, negative time is listed. Every time in between it's in positive time.
ETA: Hitting refresh hasn't made it negative.