I have always worn all Vivofit products (versions 2,3,4) while swimming. So there shouldn't be any problems with the Aquafilt.
I take off the Vivofit (and also the waterproof wristwatch) for showering, as soapy water can penetrate more easily due to the reduced surface tension.