NxtUnit

[burti]

Hi.
I programmed a version of JUnit for the nxt. So you can write tests for the nxt and run them on the nxt.
Unfortunately, it is not so comfortable like JUnit as you need to wrap the test methods' calls into objects because nxt does not offer java reflection.

I use NxtUnit all the time while programming additional basic collections for nxt (sets, maps).
You can try it out by downloading the source from http://www.cip.ifi.lmu.de/~mirwaldt/data/nxtunit.zip,
embedding it into a eclipse project and running Main in the "sample" package.

Please do not commit it to the lejos repository yet. Sven will do it for me when it is ready to be contributed.
He will also add my collections (sets, maps) after having reviewed them.
I will need to do some minor changes to NxtUnit and improve the small API to it.
So the development and usage of NxtUnit has not been finished yet.
If you have some proposals or remarks, just tell me.

Hope you enjoy it,

Michael

[burti]

a short status update for you (the team members) and me:

I discussed NxtUnit with Sven - he would enjoy to have nxtunit in lejos as I do and use it for some of his method classes.
I have used the current nxtunit-version for testing the collections I programmed. Have a look on the status of them in this post:
http://lejos.sourceforge.net/forum/viewtopic.php?f=7&t=2655
I want to point out I do not expect everybody of you in the team always to use my nxtunit and write lots of tests but I would really enjoy it whenever you use it even if it is only rare.

However things have to be improved in nxtunit:
1.) I aim at showing line number, method name and class name where an AssertionError occured instead of showing class number, method number and program counter where you need to look in the linker output and class files to resolve them into useful human-readable information.
This can be done with the debug info file. This file is atm too large to be transferred to the nxt (more than 50KB) but could be easily compressed into a new file format that decreases the size to less than 10KB. Thus nxtunit and thrown exceptions could read out those information from that file. I would do that job after my exams as I promised that to Sven.
2) The wrapper for the calls of the test method must be manually be written which is quite ugly and uncomfortable. A generator for that task exists now and is based on java reflection and regular expressions and works but has not been published yet. Sven does not like the approach I thought of and has his good reasons for that. He prefers a bcel solution. I will try out bcel after my exams and try out his ideas.

As I mentioned, I need to learn for my exams so that I cannot do all that now.
My exams will be over after 7th of June. After that I would enjoy to join your team if you accept me.
I know I'm still a stranger to some of you, so if you want to get to know me then I'm available over Skype for you after my exams.
Just contact me then if you want. I would enjoy it.

I would really appreciate joining your team, contributing to lejos and improving it if you let me.

Michael

[gloomyandy]

Hi,
I'm not sure that transferring the debug information to the NXT makes that much sense. This would require programs to contain code in the exception handler to decompress the information and would also require users to transfer the extra data to the NXT and keep it up to date just on the off chance that the program throws an unexpected exception. Then there would be the issue of how to display the (often rather long) class, and method names on the NXT display. I think it makes much more sense to provide tools on the PC side to help with the decoding of the information (which is what we already do with the remote console), we have plans to expand on this possibly before the 0.9 release...

Andy

[skoehler]

Hi,
I'm not sure that transferring the debug information to the NXT makes that much sense. This would require programs to contain code in the exception handler to decompress the information and would also require users to transfer the extra data to the NXT and keep it up to date just on the off chance that the program throws an unexpected exception. Then there would be the issue of how to display the (often rather long) class, and method names on the NXT display. I think it makes much more sense to provide tools on the PC side to help with the decoding of the information (which is what we already do with the remote console), we have plans to expand on this possibly before the 0.9 release...

For normal programs, everything you say is true. However, if you want to run a test suite, you normally don't assume that there are no exceptions. Actually, you expect some exception to happen, and you want to know where exactly it happened. That said, I believe it's much more practical, if a test suite that is being run on the NXJ actually generated nxjconsole-friendly output, however that can be achieved. Then, the testsuite can have make all sort of output, and exception traces are translated on the fly. Sounds pretty good to me.

[burti]

Hi,
I'm not sure that transferring the debug information to the NXT makes that much sense. This would require programs to contain code in the exception handler to decompress the information and would also require users to transfer the extra data to the NXT and keep it up to date just on the off chance that the program throws an unexpected exception. Then there would be the issue of how to display the (often rather long) class, and method names on the NXT display. I think it makes much more sense to provide tools on the PC side to help with the decoding of the information (which is what we already do with the remote console), we have plans to expand on this possibly before the 0.9 release...

Andy

Hi Andy,
as Sven pointed out I did not thought of transferring the debug info file with EVERY program but only with those using nxtunit and only those!
Therefore only programs "under test" will use it for reasons of comfort because they are very likely to throw exceptions relatively assertionerrors as you might agree.
The code for reading out that information will be hidden to the programmer because nxtunit will do that job.

Of course, you could not know that what Sven and I have wrote you here after your last reply in this thread.
I hope that explanations help you understand our ideas.

For the problem of limited screen size:
What about programming a kind of textarea component for longer outputs?
I would do that after my exams so that the restriction with the limited width of the screen for texts would not disturb any longer.

Michael

[gloomyandy]

I still don't think it makes sense to do this on the NXT. As Sven said it makes far more sense to me to make use of a PC side program to view the output. Trying to view full exception data on the NXT LCD screen just sounds like a pain to me...

Andy

[burti]

I still don't think it makes sense to do this on the NXT. As Sven said it makes far more sense to me to make use of a PC side program to view the output. Trying to view full exception data on the NXT LCD screen just sounds like a pain to me...

Andy

First I thought like that, too.
However I programmed a small GUI which allows you to show you failed expectations (i.e. "assertEquals(...) fails")
Displaying is actually not the problem for me.
The problem is the location of a expectation in the source that failed.
ATM you only get the class number, the method number and the program counter where it occured.
You always need to look them up to find out the location of the failed expectation.
That's terribly bothering and makes nxtunit unusable.
If you see problems in displaying error information, then I can assure you that is not such a problem as you might think.
If you regard transferring all that debug information onto the nxt as a silly thing, then I can fully agree with you. The memory of the nxt is very limited and transferring debug information to the nxt is like flooding a harddisk.
I aim at keeping those debug information on the PC in future.
Therefore Sven and I actually plan for the future that nxtunit tests should be comfortably called from eclipse like junit tests, transferred to the nxt, be run there, send their results back to eclipse and then the reults should be shown in eclipse. That would be really nice I think.
That however is not as easy as just writing a program with tests and the help of nxtunit, transferring it to the nxt and running it there (and thus the tests) on the nxt with a gui.
I just want to give all team members the chance to test their code full- or semi-automatic with nxtunit (if they want). That's all

Michael

[gloomyandy]

Hi Michael,
I'm sure that what you are doing will be very useful to the team. I just think that time spent on NXT gui displays of exceptions, or compressed debug data files, is not time well spent (though it is your time so obviously feel free to use it as you wish!). I think having a program that runs on the PC that shows the results and handles decoding any exceptions is the way to go (and there is already code to do some of this in the remote console) and is the area that should be worked on, though I hope you also try and ensure that this can be used by other IDE users and not just with Eclipse...

Anyway good luck with your project

Andy

[burti]

though I hope you also try and ensure that this can be used by other IDE users and not just with Eclipse...

Elipse was just an example. I assume it will also work with netbeans and every IDE who can handle junit-tests.

Anyway good luck with your project

Thank you!

Michael

[kvols]

I think we may be talking about different things here, leading to some confusion...

I have successfully been using JUnit for LeJOS classes on my PC (under Eclipse, but could be used in any environment). The unit tests would do a thorough test of an isolated part of a system (a 'unit', eg. an algorithm, one class that is designed to be tested independent of the rest of the system), to see if it works under some very controlled conditions (ie. the unit test).

This works very well indeed. The unit test can be run at any time, and can be part of a build procedure when compiling a program, since the unit test does not depend on anything but the unit test itself (as it should be!). I have all the information I need in my programming environment, such as line numbers, reflections, the editor etc.

Why would I run unit tests on the NXT brick instead? If the answer to that question is "That is where you have the motors and sensors", we're probably not really talking about unit tests, but some perhaps some sort of integration test: Your system (the robot) needs to be manually put into some specific state (eg. at the start of a test track) for the test to be executed. Integration testing is perfectly OK, but it is not the same as a fully automated, independent, self-contained unit test.

[burti]

Hi kvols.

Why would I run unit tests on the NXT brick instead?

GIven the case you use a collection in your unit test:
1. JUnit and JUnit tests can only use the JDK collections when you run them on the PC but your applications for the nxt can only use the lejos ones.
The general problem with using JUnit is: you need to mix the system libraries of lejos with the jdk system libraries in the same classpath with the same names in the same packages which can only fail!
NxtUnit addresses this problem: You only use the lejos libs (the right libs in opposite to the JDK libs) for your tests and run them on the nxt which is the right environment in opposite to your PC.
2. The VM on the nxt quite differs a lot from the VM by Sun/Oracle on your PC:
A collection/program can perform well on your PC but very bad on the nxt but you would not notice that when you run or test it only on your PC.
Keep in mind: Many collections are programmed in a way so that they have a good performance (HashMap, HashSet).
That's why I prefer tests on the nxt instead of the PC.

Whenever you use JDK collections in your units you cannot automatically expect it to work on the nxt the same as lejos does not support all collections from the jdk and the implementations can differ from those of the JDK.
Some operations of collections are not supported yet, others have not been (automatically) tested and can have flaws despite hard work.
The JDK implementations often are memory-consuming while lejos collections must be memory-saving and also efficient.

To sum it up, I would not trust your JUnit test too much when it uses JDK collections and runs on a PC with losts of resources (memory, cpu) though it should actually use the lejos libs and run in an embedded system with limited resources (memory, cpu) like nxt ...

Michael

[burti]

update for you and myself:
1) I have programmed a new small file format for the debug info which takes less memory than the original one.
Problem: The dump file with all the debug info takes a lot of memory. E.g. the debug info file of the lejos startup program takes 96KB.
Solution: The new file format stores name tokens like the "java" in the full class name "java.lang.Boolean" only once instead of several times and uses address registers for faster access for name resolution.
Debug info files using the new file format take less memory. E.g. the debug info file of the lejos startup program takes only 35KB using the new file format.
TODO: A) I will write a RandomAccessFile-Reader which allows to read a file from random positions. This is necessary for using the new file format efficiently, i.e. without the need of loading the all or big parts of the file into RAM.
B) I will write a StreamReader-version of the file format class so that it can resolve class numbers and method numbers to names and program counters to line numbers whithout reading all of the file.

2) I have programmed a byte code generator with bcel so that nxtunit tests are automatically embeded into the nxt unit framework i.e. wrapper classes for calling and naming the test methods are generated.
Problem: Lejos does not support Java Reflection which forces the test programmer to create wrapper objects for calls on test methods which is quite bothering and ugly.
Solution: A generator creates wrapper classes so that test programmers can focus on writing tests.
Unfortunately it generates a wrapper class for each test method which is a flaw considering the 255-classes-limitation.
TODO: I will write a new generator which will use only one wrapper class for each test class instead of each test method.