I do not fully understand how the M1 and M2 pins on the motors work, but they use pulse-width modulation (PWM) signals and support 4 states: forwards, backwards, brake and float. There is information on this in Chapter 3 of the Extreme NXT book.
The code that drives these pins is in the separate AVR processor. If you want to see the code it is in the ATmega48 directory in the downloadable LEGO MINDSTORMS NXT Firmware Open Source.
Neither leJOS nor the standard Lego firmware has direct access to these pins. Its interface is via messages send from the ARM chip to the AVR (Atmega48) processor. The information sent for each motor is the power (-100 to +100) and 1 bit specifying brake or float. It is also possible to specify the PWM frequency in this message. leJOS uses the same value as LEGO - 8khz. The leJOS code that sends the message is in nxtvm/platform/nxt/nxt_avr.c.