Streaming Sound question

[BitFlip]

I've been looking around the forum for a while and it has helped me immeasurably but I have recently come up against a problem that I cannot figure out and that I have not seen anywhere on the forums. Here goes!

I am trying to 'stream' music to the NXT brick from a PC using bluetooth. All is going well with transferring the data, but I am having a heck of a time playing it. I looked into the .wav file format and the Sound.playSample() method so I understand how to strip the .wav into pure digital sound data on the PC side. The problem comes in getting the NXT to actually play the sound data. It appears that there is a 'native' method which handles the actual playing of the sound data which I have not been able to track down in the classes or documentation. I imagine that the method will do something like take in one or more bytes of information and translate those into a number which is then used to generate an appropriate voltage at the speaker. I need to know how to access this native method and, if possible, where to locate documentation for it.

If it is relevant, I am using Lejos 0.85 on a Windows XP Home system.

Thanks for making this an excellent source of useful information about Lejos. I eagerly await your responses.

[gloomyandy]

Hi,
There is no native method to do what you want to do. The only method currently available plays a series of samples from a "file" in flash memory. So the only way to play a stream would be to write it to flash first. Unfortunately writing to flash is a tricky thing to do and tends to interrupt sound playback... What exactly are you trying to do? How long are the streams you want to play?

Even if there was a native method to play a sample from memory, I'm not sure that you would be able to get continuous playback, because of the problem of blending one set of samples into another... Also there is the isue of ensuring that any ram based buffer is correctly protected from the garbage collector... It's an interesting problem though. I'll try and take a look at adding a method to allow playback of ram based samples and see if I can get them to blend correctly...

Andy

PS The actual playback mechanism is rather more complex than simply setting a voltage for output. The NXT does not have a DtoA converter to do this. Instead it has a single bit output pin and uses a technique called Pulse Density Modulation to actually create the output waveform...

[BitFlip]

Thank you for your prompt reply, GloomyAndy.

I believe that I understand what you're saying about the PDM (I read that part and shook my head... I would love to have an DtoA converter instead as I'm more familiar with those - I understand the reasoning behind using something less expensive, though). If I am reading your message right, Lejos does not currently have the capability of adjusting the PDM fluctuation sent to the speaker; this is handled internally. If I am incorrect in this, please let me know.

A little more about the project: I am trying to stream ~10 - 30 seconds of music (8 bit wav file sampled at at least 8000 samples per second) to the brick from the PC. Unfortunately, the RAM won't hold that amount of data all at one time, so I needed some way of concurrently playing the sounds, deleting played portions of the sound, and transferring unplayed portions of the sound file to the Brick. I've looked into the BT transfer speeds described here and I think that I can make it work speedwise, but getting the threads to play nice may be another issue entirely.

If there is a way to adjust the PDM bit using Lejos, please let me know and I will experiment with it. If you can think of any other way to allow me to successfully play (audible) 30 second sound files without trying to find a workaround, also let me know.

Again, thank you for your time. Getting the opportunity to discuss technical issues with the people who actually create the software and the documentation sure trumps trying to guess how it works on my own

[skoehler]

Hi,
Even if there was a native method to play a sample from memory, I'm not sure that you would be able to get continuous playback, because of the problem of blending one set of samples into another... Also there is the isue of ensuring that any ram based buffer is correctly protected from the garbage collector... It's an interesting problem though. I'll try and take a look at adding a method to allow playback of ram based samples and see if I can get them to blend correctly...

AFAIK, the usually way to implement the blending from set of sample into another set of samples is to use ringbuffer. The ringbuffer is required to be continuously filled by the application. The OS or the hardware simply continuously plays back the data from the ring buffer.
If the ring buffer is not filled in time, some strange skipping effects may occur.

I wonder, if leJOS is capable of playing sound as a kind of background task, while the Java threads continue to run.
Also the GC may block the application for too long, and the ring buffer may become empty too fast leading to skipping in sound.

[gloomyandy]

Hi Sven/Folks,
A little more background on the way things currently work and what may need to change...

There currently is no ring buffer, but it may be that one will be needed (or at least some form of additional buffering). The high level samples do not directly drive the hardware. Each high level sample (8 bits in this case), is expanded into 256 bits of PDM data. This data is written to one of two 64 word buffers (the buffers are used alternately) and the buffer is written to the hardware using DMA. So each low level buffer contains 8 high level samples. At a sample rate of say 8KHz this means that a buffer will take 1mS to play. So we have that long to ensure that the next set of data is available. Given the 2mS timeslice used by leJOS I think it will be hard to ensure this without providing additional buffering. No buffering is currently used as the samples are simply played directly from flash (and so are always available). A relatively small ring buffer (say 256 bytes), would provide an additional 32mS of time which should be enough. Even at a sample rate of 16KHz (which is probably as high as it makes sense to go), we would still have 8mS of time to provide the data which should be enough.

The leJOS GC has a very small pause time for threads that are not performing memory allocation (less than 3mS), so hopefully this will not be a problem. We will however have to ensure that the ring buffer is protected from the GC (so that it is not moved/freed)...

All the best

Andy

[BitFlip]

GloomyAndy,

I understand where you're coming from with the need for a circular buffer to play the sound samples correctly from RAM. Thank you for providing more in-depth information on how the internals of Lejos work.

What I had originally planned to do was to to implement a large circular buffer and load it with the PCM data. I was hoping to find a low-level method or command to directly drive the speakers with the data (as you mentioned, the data must be transformed first to be compatible with the method of driving the speakers). From your previous post, it appears that I will have to change the sound buffer in the VM if I am to make this work.

I would like to know if there is a low-level or VM-level command to move information to the sound buffers (or the memory address to write to). As it appears that I may have to change the sound buffer in the VM, I would also appreciate it if you could point me towards the file(s) where I can find it.

Thanks.

[gloomyandy]

Hi,
The ring buffer will almost certainly have to be in the firmware (or at least available directly to the firmware at interrupt time). I don't think it will be possible to do that part in Java due to the scheduling/timing issues I mentioned above. The code that accesses the sound hardware (and that will need to be changed to allow access to RAM based data and add the ring buffer), is here:
http://lejos.svn.sourceforge.net/viewvc/lejos/trunk/nxtvm/platform/nxt/sound.c?revision=3118&view=markup

If you want I could take a look at adding code to support this operation and make a build available for you to try. I may not be able to make these changes until the weekend though...

Andy

[BitFlip]

Going to these lengths to address my problem is a bit unexpected and very much appreciated, GloomyAndy. This is a side project of mine and so I have plenty of time to get it done (most certainly, I can wait until the weekend!). In the mean time, I will look at the file you linked.

Thanks again, Andy.

[gloomyandy]

Hi,
It is an interesting idea and I've been meaning to take another look at the sound driver sound of things, so I'm happy to try and help! Should be fun I think, and being able to play any length of audio clip (from a PC) would be pretty cool!

All the best

Andy

[gloomyandy]

Hi Bitflip,
I sent you some email with details of my progress on this. Did you get it?

Andy

[gloomyandy]

Hi,
The code to allow PCM samples stored in a RAM buffer have now been checked into the latest svn and developers snapshot...

Andy