How to build new libraries/programs for leJOS target

A place to discus the development of leJOS for the EV3. Please do not use this section to post questions about how to use leJOS or to report problems etc.
Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

How to build new libraries/programs for leJOS target

Postby Parapampa » Sun Nov 22, 2015 4:59 pm

Hello,

We would like to have mjpeg-streamer running on the EV3 brick to stream video. We indeed want to check its performances in real time, since we had so far disappointing results with existing solutions (resolution and fps far too low).

The first step was to install the Codesourcery compiler and checkout the EV3 sources (we are doing so on a 64 bits 17.1 Linux Mint). To validate that it is working, we compiled a HelloWorld example in C that worked properly on the EV3.


The second step was to try something more serious: we tried to build native/OpenCV and native/Webcam using the included scripts. We get the following errors:

Code: Select all

./build_opencv.sh

-------------------------------------------------------------------------------
BUILDING
-------------------------------------------------------------------------------
(...)
-- Found ZLIB: /usr/lib32/libz.so (found suitable version "1.2.8", minimum required is "1.2.3")
(...)
[  2%] Building C object 3rdparty/libtiff/CMakeFiles/libtiff.dir/tif_pixarlog.c.o
/home/parapampa/dev/lejos-ev3-sdcard/native/opencv/opencv-3.0.0/3rdparty/libtiff/tif_pixarlog.c:93:18: error: zlib.h: No such file or directory
(...)


Code: Select all

./build_webcam.sh                 

-------------------------------------------------------------------------------
BUILDING
-------------------------------------------------------------------------------

/home/parapampa/dev/lejos-ev3-sdcard/native/webcam
In file included from EV3VideoImp.c:5:
lejos_internal_ev3_EV3Video.h:2:17: error: jni.h: No such file or directory



Is there any documentation explaining how to configure the environment that we have missed?
How do you do it?

Thank you in advance!

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Sun Nov 22, 2015 5:26 pm

Hi,
sorry there is no magic to this you just need to work through the problems that you hit. The leJOS build scripts are probably not very general and you may need to tweak them. So for instance in the case of the webcam it looks like your Java install is probably in a different location to mine so the line:

Code: Select all

arm-none-linux-gnueabi-gcc -shared -o libev3video.so EV3VideoImp.c -fpic -I/usr/lib/jvm/java-7-openjdk-i386/include -std=c99 -Wall 
cp *.so ../bin


Will need to be changed to specify the location of your jdk include directory.

As to the openCV problem I'm not really sure what is going on. The build system for openCV is rather complex (using cmake to create the make files) and it does not always seem to work very well when cross compiling. I suspect that what is happening is that it is finding an installation of zlib on your linux system rather than using the one that is included in the openCV source, this is not a good idea as your local copy is unlikely to be for an arm/EV3 system! So you may need to modify the cmake rules to tell it not to search your local system for things. In general when cross compiling you want to have a system with as few packages installed on it as possible. So you may find that using a basic Linux system in a VM is a better base for cross compiling things. A tip for working with this sort of build system is to use the leJOS build system to generate the files, but then to edit it so that it does not run the clean/cmake stages. You and then simply run the make step. I tend to try and fix the problem at the make level and once I have understood what the problem is I then work backwards and try and change the cmake configuration so that it will generate the "fixed" make file.

Sorry if this all sounds rather vague but that is how it is.

Going back to your original problem what exactly is it you are trying to do? If what you want to do is to use mjpeg-streamer to stream a captured video stream then you can do this using openCV in Java. Have you tried doing this? If you have what problems did you hit, you may be trying to fix the wrong problem. My experience has been that the limiting factor with streaming video is the USB bandwidth. It will be used to both capture the Video image from the camera and send the encoded image out via wifi. Unfortunately it is a USB 1.1 implementation and there simply is not much bandwidth available. If you provide more information about exactly what it is you are trying to do, what camera you are using, what frame size, what connection type etc. we may be able to help....
leJOS news https://lejosnews.wordpress.com/

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Sun Nov 22, 2015 5:31 pm

Oh and you should probably take a look at Lawrie's article about web streaming video, which has recently been updated to make use of the version of openCV that we ship with leJOS 0.9.1.
https://lejosnews.wordpress.com/2015/09 ... streaming/
leJOS news https://lejosnews.wordpress.com/

Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

Re: How to build new libraries/programs for leJOS target

Postby Parapampa » Sun Dec 06, 2015 2:23 pm

Hello gloomyandy,

Thanks for your quick and comprehensive answer!
We were hoping we had missed an automagical solution to setup the environment properly for cross-compilation. We are probably gonna explore a chroot-like solution to work around the environment conflicts.

We indeed haven't tried an OpenCV solution with leJOS yet since it is so recent and mjpeg-streamer is supposed to work out of the box (and we have used it on standard Linux boxes).

We might have overestimated our webcams' quality and they look to be our current blocking point. One of them only supports YUYV while the other is also supposed to offer MJPEG - we did not manage to make that work with the simple java http server built by Lawrie. That is why we then decided to try a standard solution for the webcam.
Anyway, we just invested some money in the Logitech C920 in order to try playing with H264, which should give better results, especially considering the restrictions we have with usb 1.1.

In the meantime, we will consider OpenCV in Java since it should make compiling foolproof :).

Thanks again. We hope we will have interesting stuff to share here quite soon!

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Sun Dec 06, 2015 5:02 pm

If you have a camera that produces mjpeg and all you want to to is stream the images (and you do not want to access them on the EV3), then you should be able to simply use the leJOS webcam class to capture the mjpeg stream and you can then use a modified version of Lawrie's web server code to stream it (without having to decode/reencode back mjpeg). I've been able to do this and was able to stream 640x480 images at around 20fps. Note that you will need to use 0.9.1 to do this as earlier webcam classes did not allow you to specify the image format. You may need to list the various formats supported by your camera so that you can specify the correct size/format to get the camera to generate mjpeg. You can do this by using the v4l-ctl command that is included in the latest version of leJOS.
leJOS news https://lejosnews.wordpress.com/

Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

Re: How to build new libraries/programs for leJOS target

Postby Parapampa » Sun Jan 10, 2016 5:55 pm

Hello GloomyAndy,

Thanks for your answer, quick and to the point as always!

We have tried cross-compiling with some moderate success, that is being able to use mjpeg-streamer on leJOS to get some nice stream at 640*480 with around 15/20fps.

Using leJOS 0.9.1, we did not manage to get such result, to say the least. We indeed more got something around 20 sec per frame D:. And we used the classes provided with leJOS:

Code: Select all

Video video = ev3.getVideo();
video.open(WIDTH, HEIGHT, Video.PIX_FMT_MJPEG, Video.FIELD_ANY, FPS);


Have you handled differently on the brick or client side to get the good results you mentioned?

Appart from that, we have our h264 camera, which format and resolutions are well supported as seen through v4l2-ctl. Yet, we are so far unsuccessful in managing to stream directly it to a client. Work in progress...

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Sun Jan 10, 2016 6:23 pm

I think you need to post a little more of your code. You really should be able to get around 15fps in Java (that was what I was getting), but you must ensure that you ask for a format that support mjpeg and obviously should not decode or encode it, simply pass it on.
leJOS news https://lejosnews.wordpress.com/

Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

Re: How to build new libraries/programs for leJOS target

Postby Parapampa » Sun Jan 10, 2016 6:44 pm

Ok, so the whole code on the brick is pretty simple:

Code: Select all

    private static final int WIDTH = 640;
    private static final int HEIGHT = 480;
    private static final int NUM_PIXELS = WIDTH * HEIGHT;

        EV3 ev3 = (EV3) BrickFinder.getLocal();


        Video video = ev3.getVideo();
        video.open(WIDTH, HEIGHT, Video.PIX_FMT_MJPEG, Video.FIELD_ANY, 30);

        byte[] frame = video.createFrame();

        Socket sock = new Socket(PC_HOST, PORT);
        BufferedOutputStream bos = new BufferedOutputStream(sock.getOutputStream());


        while(Button.ESCAPE.isUp()) {
            try {
                video.grabFrame(frame);
                bos.write(frame);
                bos.flush();
            } catch (IOException e) {
                break;
            }
        }


That's about it! Any thought?

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Sun Jan 10, 2016 6:55 pm

You need to use the return from grabFrame as the number of bytes to write. At the moment your code will be writing an array of bytes that it sized for the maximum possible frame size, rather than the actual compressed frame size... So you will need to use a write method that allows you to specify the size.
leJOS news https://lejosnews.wordpress.com/

Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

Re: How to build new libraries/programs for leJOS target

Postby Parapampa » Sun Jan 10, 2016 7:48 pm

Ok, this must be the part we were looking for! We were indeed wondering how this could be true MJPEG with a static byte[] size. What is disturbing is that we had no issues having coherent images on the server's size; we would have expected the stream to be all corrupted...

We'll give it a shot and post back here. Thanks again for your helpful input!

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Sun Jan 10, 2016 8:32 pm

The stream would be fine. The mjpeg part will say how long the usable data is and you will just be sending a bunch of extra data over that is then ignored by the reader. At least that is my guess.
leJOS news https://lejosnews.wordpress.com/

Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

Re: How to build new libraries/programs for leJOS target

Postby Parapampa » Mon Feb 15, 2016 11:02 am

Hello gloomyandy,

Your last answer was helpful in the way it enabled us to get a mjpeg streaming in java. Thank you for that, this was an interesting test!
Unfortunately, the FPS count was still too low (around 9) for our needs, and the resolution was not good enough for our purpose (even after killing all non-needed processes on the brick).

So, we are back to square one and have been toying a lot with gstreamer to:

    1- Find a working pipeline (using tcp for streaming from the brick), and parsing / decoding h264 on a client PC
    2- Compile it for the brick target

We managed to have 1- working on a PC, and 2- was successful too, since we can start gstreamer on the brick.
Unfortunately, despite having some success with YUYV, we failed with H264...

Many headaches later, we stumbled on the root of the issue: the videodev2.h header (/usr/include/linux/videodev2.h) provided by leJOS is too old (2007), and we would need a more recent one for H264 (like the one from 2012). A web article http://www.oz9aec.net/index.php/gstreamer/473-using-the-logitech-c920-webcam-with-gstreamer confirms this:

First, you need Linux kernel 3.2 or later to have "H264 pixel format" supported in the v4l2 drivers. I heard some people talk that earlier kernels can be patched to work – I do not know, nor do I care wish to explore that option. AFAIK most recent Linux distributions come with Linux 3.2 or later so it shouldn't be a problem.


This means we hit a wall :? . For now... :)

Do you know if an upgrade of the kernel is scheduled anytime soon? Or would you have pointers to provide us to update this header?

We strongly believe native h264 support in leJOS would benefit many people looking for a decent working real-time streaming solution, and should we manage to have the correct header included, we would then gladly share with the community the steps to have gstreamer work on an EV3 leJOS brick (with github code, pipelines, and compile steps)!

PS: if you think we might start getting of-topic here, please do not hesitate to ask me to have this discussion somewhere else on the forum.

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Mon Feb 15, 2016 4:56 pm

Why exactly do you need to do this? If all you want is a video stream why not simply use a WiFi based camera and have done with it? Seems pointless using a load of resources on the EV3 if you aren't going to process the data on the brick at all.

If you do want to continue down this route then why not give ev3dev a try? That way you will know if what you are trying to do is even possible or not.
leJOS news https://lejosnews.wordpress.com/

Parapampa
New User
Posts: 18
Joined: Sun Nov 22, 2015 3:53 pm

Re: How to build new libraries/programs for leJOS target

Postby Parapampa » Tue Feb 16, 2016 8:06 am

Thanks for you reply.

The reasons are simple, and we considered them many times:

    1. An usb camera is cheaper than an good IP webcam with wifi.

    2. Such a camera would require an external power supply, which is cumbersome to manage and add significant weight to the robot. IP cameras are usually heavier too. This means two batteries for two wifi connections on one robot, which looks overkill.

    3. Startup of a robot would be more complex : instead of simply turning the robot on with its program, you would also have to connect the IP camera to a power source, and maybe turn it on too. This is also one more IP connection to manage.

    4. Since we plan to have many similar robots, we really want to simplify all those points for cost, durability and practicalities reasons while keeping it scalable.

You are right however: we have no guarantee we can achieve the result we target on the brick. We have reasons to believe it is yet doable because of the achivements we go with mjpeg-streamer running on it, with a decent image quality but a framerate a bit too low for us. We would really like to keep working with leJOS since the platform has been pleasant to work with so far, and besides we need the support it provides for sensors such as the RFID one (and ev3dv does not).
But then, if you advise us to do so, then we might try this with ev3dev, at least for a proof of concept.

Just to clarify: do you confirm that there is no plan to update the kernel in the coming weeks/months?

User avatar
gloomyandy
leJOS Team Member
Posts: 6057
Joined: Fri Sep 28, 2007 2:06 pm
Location: UK

Re: How to build new libraries/programs for leJOS target

Postby gloomyandy » Tue Feb 16, 2016 9:13 am

Updating the kernel is on my list of things to do. However it is a very big change (since it will almost certainly also need an update of the entire rootfs). So if it does happen it is unlikely to happen very soon. But as I said you could use ev3dev to check to see if you can get the frame rate and size that you require (which to be honest I'm not convinced you will be able to get I just don't think that the EV3 USB can handle that sort of data rate from the camera and then to the WiFi).
leJOS news https://lejosnews.wordpress.com/


Return to “leJOS EV3 Development”

Who is online

Users browsing this forum: No registered users and 1 guest