<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Projects |</title><link>https://www.ugr.es/~cvrlab/projects/</link><atom:link href="https://www.ugr.es/~cvrlab/projects/index.xml" rel="self" type="application/rss+xml"/><description>Projects</description><generator>HugoBlox Kit (https://hugoblox.com)</generator><language>en-us</language><lastBuildDate>Thu, 15 Oct 2026 00:00:00 +0000</lastBuildDate><image><url>https://www.ugr.es/~cvrlab/media/icon_hu_138e5d72c01fd19a.png</url><title>Projects</title><link>https://www.ugr.es/~cvrlab/projects/</link></image><item><title>BRAINAV - BRAIn-inspired visual processing for real-time energy-efficient autonomous NAVigation</title><link>https://www.ugr.es/~cvrlab/projects/brainav/</link><pubDate>Thu, 19 Jan 2023 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/brainav/</guid><description>&lt;ul&gt;
&lt;li&gt;This work was supported by the National Project PID2022-141466OB-I00 funded by MICIU/AEI/10.13039/501100011033 and by ERDF/EU.
&lt;figure&gt;&lt;img src="https://www.ugr.es/~cvrlab/projects/brainav/aei_logo.png"&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="summary"&gt;Summary&lt;/h2&gt;
&lt;p&gt;BRAINAV aims at developing bio-inspired pipelines for real-time energy-efficient visual navigation that operates in the real-world in optimized platforms embedded onboard mobile robots. First, we need the development of a new paradigm for computer vision that allows us to effectively work with asynchronous neuromorphic sensors that imitate visual retinas and the development of high-performance architectures for reaching real-time performance with them. Then, we also require efficient ML networks for reaching low-energy inference and eventually, to propose new bio-inspired components to solve mapping, localization, and path planning for exploration. We are committed to prove the potential of our energy-efficient vision pipelines and therefore, we propose the use case of autonomous navigation using mobile robots.&lt;/p&gt;
&lt;p&gt;Video processing is crucial in perception due for environment understanding. However, computer vision is a very demanding task and in robotics it requires to close perception-action loops in real-time. Moreover, distributed local mobile nodes need low-energy operation since they run on batteries. Power consumption vs. performance is a challenging trade-off, especially in distributed environments, given the number of sensors, the heterogeneity of the system, their autonomy or their need of high performance. Our hypothesis is that biologically-inspired strategies will help us achieve energy-efficient solutions that reach good levels of accuracy while operating in real-time in real scenarios.&lt;/p&gt;
&lt;p&gt;The final demonstrators consist of UAV platforms equipped with event and conventional cameras, inertial sensors, telemetry and GNSS, with one configuration additionally including LiDAR. All perception, planning and control components run onboard using an NVIDIA Jetson Orin. Real outdoor experiments validated autonomous flight, trajectory tracking at speeds up to 8 m/s, detection of other UAVs, autonomous obstacle perception and avoidance, and simple LiDAR-based mapping. BRAINAV contributes directly to the thematic priority &amp;ldquo;Digital World, Industry, Space and Defence&amp;rdquo; through advances in artificial intelligence, autonomous robotics, neuromorphic vision and embedded processing.&lt;/p&gt;
&lt;h2 id="objectives"&gt;Objectives&lt;/h2&gt;
&lt;p&gt;The main general objective of BRAINAV is the development of a bio-inspired pipeline for real-time
energy-efficient visual navigation that operates in the real-world in optimized platforms embedded in
mobile robots. First, we need the development of a new paradigm for computer vision that allows us
to effectively work with these asynchronous neuromorphic sensors and the development of high-
performance architectures for reaching real-time performance. Then, we also require efficient ML
networks for reaching low-energy inference and eventually, to propose new bio-inspired components
to solve mapping, localization, and path planning for exploration. Finally, we also propose to prove
the potential of our energy-efficient vision pipelines for autonomous navigation using mobile robots.&lt;/p&gt;
&lt;p&gt;Our specific objectives (SO) are detailed next:&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO1. To develop of event-driven perception for navigation tasks.&lt;/em&gt;
As mentioned before in the state of the art, some works have addressed the task previously with event-
based sensors [BAR14a, BAR21, VID18]. However, their results are still very poor, reaching only low
performance. In our previous work, we have thoroughly studied alternative knowledge-based methods
for 3D perception, mainly image motion and 3D motion estimation. In BRAINAV, we
propose to study new methods based on Machine Learning solutions [DEN23, ISE23] to achieve
better accuracy performance. Also, most of the previous methods build images out of accumulation of
events for prefixed time intervals and apply classic methods to these artificial images. This severely
harms the performance and does not take advantage of the main feature of neuromorphic sensors, their
high-temporal resolution or low latency. We propose the development and evaluation of event-driven
methods for motion and 3D geometry estimation [BAR21]. Lastly, the accuracy of our solution will be
evaluated using novel benchmarks and datasets that integrate both events and images using the motion
capture system available in our lab. Moreover, we are committed to the Open Science principles and
will release the datasets for the community. This is crucial for the adv ancement of neuromorphic
engineering, due the current lack of datasets and their relevance for Machine Learning techniques.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO2. To optimize for real-time energy-efficient processing engines.&lt;/em&gt;
In an effort to reduce the computational complexity of the prop osed solutions, we will explore
different mechanisms for low-energy computation. First, we propose the use of optimization
techniques on ANNs such as: model quantization [LIA21] shortening bit widths for inference to reduce
computational complexity; pruning [VAD22], that removes connections with smaller values in
networks; and distillation [DEN23], that allows transferring knowledge from teacher complex models
to student simpler ones. Second, we are also interested in exploring new techniques based on Spiking
Neural Networks. These are biologically-plausible neural architectures do take into account energy
consumption [DAV21] doing learning with sparse data. We also propose the exploration of brain-like
structures to build new navigation methods that imitate their functionalities.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO3. To implement real-time processing on embedded platforms with neuromorphic sensors.&lt;/em&gt;
Dealing with a real-world application imposes restrictions such as real-time performance for closing
our perception-action loops for autonomous navigation. We will consider different hardware
accelerators such as GPUs or FPGAs, using the latest advancements for the development of fine-
tuning pipelines (e.g. DeepStream) and for optimized hardware datapaths (e.g. Vitis AI). The selected
hardware accelerator will impose different non-functional requirements on the implementations
developed for SO1 and functionally optimized for SO2, for performance, latency and power
consumption. This objective also pursues the effective integration of the asynchrono us sensors into
von Neumann architectures which is by itself, an additional challenge.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO4. To demonstrate embedded autonomous navigation onboard mobile robots&lt;/em&gt;
Finally, this objective aims at the validation of the proposed models and architectures considered in
the SO1-SO3, with different demonstrators for autonomous navigation tasks in real-world
environments. This application allows us to demonstrate the maximum potential of our optimized bio-
inspired pipelines and downstream processing, while ensuring the overall optimization in the
exploration task relying mainly on learning models. Our goal in SO4 is to develop a first semi-
autonomous platform to perform first localization and mapping. This prototype will have a minimum
intervention of a human pilot, and will be used to validate the mentioned tasks in unknown scenarios.
The second prototype will show the final autonomous explorer that will be enabled with additional
downstream applications such as the detection of other mobile robots in the scenario and possibly
tracking them, or finding specific targets.&lt;/p&gt;
&lt;h2 id="final-autonomous-navigation-demonstrator"&gt;Final autonomous navigation demonstrator&lt;/h2&gt;
&lt;p&gt;The final BRAINAV demonstrator integrates the main scientific and technological results of the project into a real autonomous UAV. The platform combines:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Neuromorphic event cameras&lt;/li&gt;
&lt;li&gt;Conventional cameras&lt;/li&gt;
&lt;li&gt;Inertial sensing&lt;/li&gt;
&lt;li&gt;GNSS and telemetry&lt;/li&gt;
&lt;li&gt;LiDAR in one of the configurations&lt;/li&gt;
&lt;li&gt;NVIDIA Jetson Orin onboard processing&lt;/li&gt;
&lt;li&gt;ROS-based perception, planning and control&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All computation is performed onboard the UAV. The platform was validated through real outdoor flights at the Cubillas test area.&lt;/p&gt;
&lt;figure&gt;&lt;img src="https://www.ugr.es/~cvrlab/projects/brainav/brainav_flight_test.jpg"
alt="UAV platform and outdoor autonomous flight experiments at the Cubillas test area."&gt;&lt;figcaption&gt;
&lt;p&gt;UAV platform and outdoor autonomous flight experiments at the Cubillas test area.&lt;/p&gt;
&lt;/figcaption&gt;
&lt;/figure&gt;
&lt;h2 id="background-references"&gt;Background references&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;[BAR14a] F. Barranco, C. Fermüller, Y. Aloimonos. &amp;ldquo;Contour motion estimation for asynchronous event-driven cameras.&amp;rdquo; Proceedings of the IEEE 102, no. 10 (2014): 1537-1556.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[BAR21] F. Barranco, C. Fermüller, Y. Aloimonos, E. Ros. &amp;ldquo;Joint direct estimation of 3D geometry and 3D motion using spatio temporal gradients.&amp;rdquo; Pattern Recognition, 113, 2021.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[DAV21] S. Davidson, S. B. Furber. &amp;ldquo;Comparison of artificial and spiking neural networks on digital hardware.&amp;rdquo; Frontiers in Neuroscience 15 (2021): 651141.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[DEN23] D. Deniz, J. Isern, J. Solanti, P. Jääskeläinen, P. Hnětynka, L. Bulej, E. Ros, F. Barranco. &amp;ldquo;Efficient reconfigurable CPS for monitoring the elderly at home via Deep Learning.&amp;rdquo; Journal of Signal Processing Systems, (2023).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[ISE23] J. Isern, G. Jimenez-Perera, L. Medina-Valdés, P. Chaves, D.Pampliega F. Ramos, F. Barranco. &amp;ldquo;A Cyber-Physical System for integrated remote control and protection of smart grid critical infrastructures.&amp;rdquo;, Journal of Signal Processing Systems, 2023.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[LIA21] T. Liang, J. Glossner, L. Wang, S. Shi, X. Zhang. &amp;ldquo;Pruning and quantization for deep neural network acceleration: A survey.&amp;rdquo; Neurocomputing 461 (2021): 370 -403.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[VAD22] S. Vadera, S. Ameen. &amp;ldquo;Methods for pruning deep neural networks.&amp;rdquo; IEEE Access 10 (2022): 63280-63300.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[VID18] A. Vidal, H. Rebecq, T. Horstschaefer, D. Scaramuzza. &amp;ldquo;Ultimate SLAM? Combining events, images, and IMU for robust visual SLAM.&amp;rdquo; IEEE Robotics and Automation 3, no. 2 (2018).&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="lists-of-contributions-publications-datasets-and-code-repositories"&gt;Lists of contributions: publications, datasets, and code repositories&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;[DEN24] Deniz, D., Ros, E., Ortigosa, E. M., &amp;amp; Barranco, F. (2024). &lt;strong&gt;Optimized edge-cloud system for activity monitoring using knowledge distillation.&lt;/strong&gt; Electronics, 13(23), 4786.
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[NOV24] Novo, A., Lobon, F., Garcia de Marina, H., Romero, S., &amp;amp; Barranco, F. (2024). &lt;strong&gt;Neuromorphic perception and navigation for mobile robots: a review.&lt;/strong&gt; ACM Computing Surveys, 56(10), 1-37.
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[DEN25] Deniz, D., Isern, J., Solanti, J., Jääskeläinen, P., Hnětynka, P., Bulej, L., Ros, E., &amp;amp; Barranco, F. (2025). &lt;strong&gt;Efficient reconfigurable system for home monitoring of the elderly via action recognition.&lt;/strong&gt; Engineering Applications of Artificial Intelligence, 158, 111383.
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[JIM25] Jimenez-Perera, G., Valencia-Vidal, B., Luque, N. R., Ros, E., &amp;amp; Barranco, F. (2025). &lt;strong&gt;Informed Federated Learning to Train a Robotic Arm Inverse Dynamic Model.&lt;/strong&gt; IEEE Robotics and Automation Letters, 10(10), 11022-11029.
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[VAZ26] Vázquez, V., Valenzuela, E., Shepstone, R., Megías, C., Miccichè, G., Ros, E., &amp;amp; Barranco, F. (2026). &lt;strong&gt;Remote handling operation for IFMIF-DONES supported by time-sensitive networking.&lt;/strong&gt; Nuclear Fusion, 66, 046029.
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[NOV25a] Novo, A., Fermüller, C., Rodríguez-Álvarez, M., Romero, S., &amp;amp; Barranco, F. (2026). &lt;strong&gt;On ego-motion estimation from event-inertial-aided normal flow fields.&lt;/strong&gt; Computer Vision and Image Understanding. Under review.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[NOV25b] Novo, A., Fermüller, C., Romero, S., Rodríguez-Álvarez, M., &amp;amp; Barranco, F. (2026). &lt;strong&gt;Event-based vision for online inference of fine-grained manipulation actions.&lt;/strong&gt; Engineering Applications of Artificial Intelligence. Under review (second round).&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="other-scientific-contributions"&gt;Other scientific contributions&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;[SAL25-CONG] Salinas, I., Romero, S. F., Molero, J. A., Gutiérrez de León, B., Banqueri, J., &amp;amp; Barranco, F. (2025). &lt;strong&gt;Clasificación de UAS basado en Inteligencia Artificial para dar soporte a sistemas de defensa antiaérea.&lt;/strong&gt; XII Congreso Nacional de I+D en Defensa y Seguridad, Zamora, Spain, pp. 105-106. ISBN 978-84-1083-073-8. Premio Antonio Torres al mejor trabajo (ISDEFE).&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[HER25] Heredia, A. J., Pérez, F. V., Novo, A., Romero, S. F., García Imbernón, G., Ortega, J. F., García, V., Albaladejo, D., Méndez, M., &amp;amp; Barranco, F. (2025). &lt;strong&gt;Automatización inteligente del sostenimiento en buques mediante sensórica avanzada e IA.&lt;/strong&gt; XII Congreso Nacional de I+D en Defensa y Seguridad, Zamora, Spain, pp. 167-168. ISBN 978-84-1083-073-8.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;[CUA23] Cuadrado, J., Barranco, F., &amp;amp; Masquelier, T. (2023). &lt;strong&gt;Optical flow estimation from event- and frame-based camera fusion.&lt;/strong&gt; Biocomp Colloque 2023, Toulouse, France.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="datasets-and-open-source-resources"&gt;Datasets and open-source resources&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;IndoorActionDataset&lt;/strong&gt; — Dataset associated with [DEN24].
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;CIAV Dataset&lt;/strong&gt; — Dataset associated with [DEN25].
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Efficient action recognition models&lt;/strong&gt; — Code and neural-network weights associated with [DEN25].
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Baxter robot dataset for learning dynamic models&lt;/strong&gt; — Dataset associated with [JIM25], released under CC BY 4.0.
&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;E-MAD — Event-based Manipulation Action Dataset&lt;/strong&gt; — Dataset associated with [NOV25b]. The dataset is currently under embargo during the publication process and will be released under CC BY 4.0 after publication.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Event-inertial ego-motion code&lt;/strong&gt; — Code associated with [NOV25a]. The repository will be made public after publication of the corresponding article.&lt;/p&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="authors"&gt;Authors&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Francisco Barranco&lt;/strong&gt; - Principal Investigator, University of Granada, CVRLab / CITIC-UGR&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="acknowledgments"&gt;Acknowledgments&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;This work was supported by the National Project PID2022-141466OB-I00 funded by MICIU/AEI/10.13039/501100011033 and by ERDF/EU.
&lt;figure&gt;&lt;img src="https://www.ugr.es/~cvrlab/projects/brainav/aei_logo.png"&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="license"&gt;License&lt;/h2&gt;
&lt;p&gt;Software, datasets and other research outputs associated with BRAINAV are distributed under the licenses indicated in their corresponding repositories.&lt;/p&gt;
&lt;p&gt;Unless otherwise stated, the contents of this website are provided for scientific dissemination purposes. Please refer to each publication, dataset or software repository for its specific copyright and licensing conditions.&lt;/p&gt;</description></item><item><title>BIO-PERCEPTION - Next generation of smart vision systems for real-time processing with bio-inspired sensors</title><link>https://www.ugr.es/~cvrlab/projects/bioperception/</link><pubDate>Mon, 20 Dec 2021 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/bioperception/</guid><description>&lt;ul&gt;
&lt;li&gt;This work was supported by the AEI through the grant PID2019-109434RA-I00 / AEI / 10.13039/501100011033.
&lt;figure&gt;&lt;img src="https://www.ugr.es/~cvrlab/projects/bioperception/aei_logo.jpeg"&gt;
&lt;/figure&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="summary"&gt;Summary&lt;/h2&gt;
&lt;p&gt;The 3D perception of our environment such as the estimation of image motion, 3D motion, or the 3D structure of unknown scenes, are tasks that humans efficiently do. In fact, we do it in a precise manner enabling us to navigate around our environment and interact with complex dynamic scenarios in a natural way. Despite of the advances in robotics, computational capacity, and computer vision, the current state of the art cannot provide efficient solutions that are even close in performance. It is consequence of the impossibility of closing perception-action cycles. Every action performed causes a change in the environment that the perception system needs to sense, to implement the next action accordingly. The key factor for the success of the current smart vision systems is not only the perception, or the execution of the correct action; it is the capacity of performing the whole loop in real-time to allow the continuous interaction. Conventional solutions propose the increase of the computational complexity, without taking into account factors such as the energy consumption, required resources, or processing time. The most advanced high-performance architectures have problems dealing with the amount of information generated by high-resolution cameras fast enough for enabling dynamic agents to interact in the real world (for example, state- of-the-art solutions take minutes for image motion estimation). Moreover, processing has to be local in order to respond to the real-time
demands.&lt;/p&gt;
&lt;p&gt;Our goal for this project is precisely to set up the basis for a new generation of smart autonomous agents that are able to carry out 3D perception in real-time, using biologically-inspired vision sensors. These sensors are based on the manner visual information is processed by the human eye. Rather than transmitting all the static information continuously, the retina selects the most relevant information, comprising it to be further processed in the visual cortex. In the very same way, these sensors independently processed all pixels and only trigger changes in the scene (&amp;rsquo;events&amp;rsquo;) in the case a substantial difference in the intensity luminance happens over time for a specific location (this happens only at object contours and textures). This allows for the reduction of the transmission of redundant information and hence, the data bandwidth. The other key advantage is that these sensors reach very low latencies, of a few microseconds.&lt;/p&gt;
&lt;p&gt;The current paradigm for computation with this kind of sensors is still very basic. Usually, asynchronous events are accumulated during short prefixed time intervals to create synthetic images and then, classic methods are applied to them. Unfortunately, this causes the increase of latencies and it generates redundant information that has to be processed in the same manner systems for conventional sensors do. We propose in this project a novel paradigm for event-driven computation: 1) event-wise processing, for every event that comes in, the model is updated and the event information is integrated for a specific task, taking full advantage of the high-temporal
resolution and 2) only the relevant information is transmitted.&lt;/p&gt;
&lt;p&gt;Finally, regarding the resources we will use hardware accelerators for massively parallel processing such as GPU or FPGAs that process huge amounts of data at once, guaranteeing bounded latencies and low energy consumption. This will also enable the integration of our system via embedded platforms.&lt;/p&gt;
&lt;h2 id="objectives"&gt;Objectives&lt;/h2&gt;
&lt;p&gt;BIO-PERCEPTION is focused on the development of a new paradigm for vision systems that integrates neuromorphic event-based sensors. Additionally, we propose the validation of this new model by building smart vision systems for autonomous navigation, proposing drones for that. Our specific objectives (SO) are detailed next:&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO1. Development of a new model for event-based 3D perception.&lt;/em&gt;
As mentioned before in the state of the art, some works have addressed the task previously with event-based sensors [BEN12, BEN14, BAR14]. However, their results are still very poor, reaching only low performance. Moreover, most of these methods build images out of accumulation of events for prefixed time intervals and apply classic methods to these artificial images. This severely harms the performance and does not take advantage of the main feature of these sensors, their high-temporal resolution or low latency.
We propose the development and evaluation of a fully event-driven model, although it will also include the possibility of integrating conventional synchronous images to improve accuracy. The models for motion object contours will be applied here [FER01, BAR14], to estimate the image motion. Using the image motion, 3D pose and velocity can also be subsequently estimated [FER01, BAR16] and, independently moving objects (IMOs) detected in the scene. Lastly, the accuracy of our solution will be evaluated using novel benchmarks and datasets that integrate both events and images.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO2. Development of a new model for event-based object segmentation.&lt;/em&gt;
Although the 3D perception module provides sparse results, object segmentation is an image-based task, which implies the need for a centralized model of the scene. In SO2, our first goal is to come up with a linking mechanism to group events and be able to distinguish different depth layers (separated by contours). Our group has already developed some models such as in [BAR15b] that could not reach real-time performance or the required accuracy. Given the popularity of Machine Learning approaches and the expertise of our Department in efficient machine learning applications, we will consider the use of Deep Learning, to develop architectures based on CNNs (Convolutional Neural Networks). These networks have proven their success in this task for images [SHE15]. After separating events from contours and events generated due to the textured regions of the objects in the scene, we will group them together in sets according to the object these events belong to. We also propose the implementation of an efficient mechanism to do that. As in SO1, we will produce new benchmarks and datasets to assess the accuracy of our models, and compare conventional and event-based approaches.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO3. Real-time processing architectures with event-based neuromorphic sensors.&lt;/em&gt;
In the motivation we already stated that autonomy cannot be reached until perception-action loops are closed in real-time. This means that the visual perception processing should be fast enough to allow agents to interact with the dynamic real-world. Moreover, in order to embed the visual processing in our autonomous agents, we should consider platforms to integrate our processing architectures on. In order to do that, we propose the use of hardware accelerators. Our group has long expertise on optimized-CPU implementations, and GPUs, or FPGAs [BAR14b, NAV17]. Also, we will consider integrated solutions with the existing UAV platforms and their embedded processors. Models and implementations designed according to SO1 and SO2 will be adapted to the non-functional requirements related to performance, latency, or memory imposed by the selected architecture.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;SO4. Demonstrator: Autonomous drone platform.&lt;/em&gt;
Finally, we also consider that the validation of the proposed models has to be shown in the framework of autonomous navigation tasks. This application allows us to demonstrate the maximum potential of the neuromorphic sensors, specifically regarding the high-temporal resolution and low latency, for example with high-speed maneuvers. Our goal in SO4 is to develop a first semi-autonomous platform that shows navigation with a minimum intervention of a human pilot for the first case, and that validates tasks such as selection of targets, segmentation, or detection of independently moving objects while navigating. The second prototype will show the whole system working to do tasks completely autonomously, without human intervention, while navigating avoiding obstacles.&lt;/p&gt;
&lt;h2 id="bibliography"&gt;Bibliography&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;[BAR14a] F. Barranco, C. Fermüller, Y. Aloimonos. &amp;ldquo;Contour motion estimation for asynchronous event-driven cameras.&amp;rdquo; Proceedings of the IEEE 102, no. 10 (2014): 1537-1556.&lt;/li&gt;
&lt;li&gt;[BAR14b] F. Barranco, J. Diaz, B. Pino, E. Ros. &amp;ldquo;Real-time visual saliency architecture for FPGA with top-down attention modulation.&amp;rdquo; IEEET on Ind. Informatics 10, 3, 726-1735, 2014.&lt;/li&gt;
&lt;li&gt;[BAR15] F. Barranco, C. Fermuller, Y. Aloimonos. &amp;ldquo;Bio-inspired motion estimation with event-driven sensors.&amp;rdquo; In International Work-Conference on Artificial Neural Networks, pp. 309-321, 2015.&lt;/li&gt;
&lt;li&gt;[BAR16] F. Barranco, C. Fermuller, Y. Aloimonos, T. Delbruck. &amp;ldquo;A dataset for visual navigation with neuromorphic methods.&amp;rdquo; Frontiers in neuroscience 10 (2016): 49.&lt;/li&gt;
&lt;li&gt;[BEN12] R. Benosman, S. Ieng, C. Clercq, C. Bartolozzi, M. Srinivasan. &amp;ldquo;Asynchronous frameless event-based optical flow.&amp;rdquo; Neural Networks 27 (2012): 32-37.&lt;/li&gt;
&lt;li&gt;[BEN14] R. Benosman, C. Clercq, X. Lagorce, S. Ieng, C. Bartolozzi. &amp;ldquo;Event-based visual flow.&amp;rdquo; IEEE transactions on neural networks and learning systems 25, no. 2 (2014): 407-417.&lt;/li&gt;
&lt;li&gt;[FER01] C. Fermüller, D. Shulman, Y. Aloimonos. &amp;ldquo;The statistics of optical flow.&amp;rdquo; Computer Vision and Image Understanding 82, no. 1 (2001): 1-32.&lt;/li&gt;
&lt;li&gt;[NAV17] F. Naveros, J. Garrido, R. Carrillo, E. Ros, N. Luque. &amp;ldquo;Event-and time-driven techniques using parallel CPU-GPU co-processing for spiking neural networks.&amp;rdquo; Front. in Neuroinf. 11 (2017): 7.&lt;/li&gt;
&lt;li&gt;[SHE15] W. Shen, X. Wang, Y. Wang, X. Bai, Z. Zhang. &amp;ldquo;Deepcontour: A deep convolutional feature learned by positive-sharing loss for contour detection.&amp;rdquo; CVPR, pp. 3982-3991. 2015.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="lists-of-contributions-publications-datasets-and-code-repositories"&gt;Lists of contributions: publications, datasets, and code repositories&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Isern, J., F. Barranco, D. Deniz, J. Lesonen, J. Hannuksela, R. R. Carrillo. &lt;strong&gt;Reconfigurable cyber-physical system for critical infrastructure protection in smart cities via smart video-surveillance.&lt;/strong&gt; Pattern Recognition Letters 140 (2020): 303-309.
&lt;/li&gt;
&lt;li&gt;Déniz, D., G. Jimenez-Perera, R. Nolasco, J. Corral, F. Barranco. &lt;strong&gt;Deep multimodal habit tracking system: A user-adaptive approach for low-power embedded systems.&lt;/strong&gt; Journal of Signal Processing Systems (2023),
&lt;/li&gt;
&lt;li&gt;Isern, J., G. Jimenez-Perera, L. Medina-Valdés, P. Chaves, D. Pampliega, F. Ramos, F. Barranco. &lt;strong&gt;A Cyber-Physical System for integrated remote control and protection of smart grid critical infrastructures.&lt;/strong&gt; Journal of Signal Processing Systems (2023),
&lt;/li&gt;
&lt;li&gt;Deniz, D., J. Isern, J. Solanti, P. Jääskeläinen, P. Hnětynka, L. Bulej, E. Ros, F. Barranco. &lt;strong&gt;Efficient reconfigurable CPS for monitoring the elderly at home via Deep Learning&lt;/strong&gt;, submitted to Engineering Applications of Artificial Intelligence, 2022.
&lt;/li&gt;
&lt;li&gt;Cuadrado, J., U. Rancon, B.R. Cottereau, F. Barranco, T. Masquelier, &lt;strong&gt;Optical flow estimation from event-based cameras and spiking neural networks&lt;/strong&gt;, Frontiers in Neuroscience, 17, 1160034, (2023)
&lt;/li&gt;
&lt;li&gt;Deniz, D., C. Fermuller, E. Ros, M.Rodriguez-Alvarez, F. Barranco. &lt;strong&gt;Event-based vision for early prediction of manipulation actions&lt;/strong&gt;, submitted since January 2023 to IEEE T. on Pattern analysis and machine intelligence.
&lt;/li&gt;
&lt;li&gt;Deniz, D., E. Ros, E. M. Ortigosa, F. Barranco. &lt;strong&gt;Optimized edge-cloud system for activity monitoring using knowledge distillation&lt;/strong&gt;, submitted June 2023 to IEEE T. on Mobile Computing.&lt;/li&gt;
&lt;li&gt;Novo, A., F. Lobon, H. G. de Marina, S. Romero, F. Barranco. &lt;strong&gt;Brain-Inspired Autonomous Robotic Navigation: A Survey&lt;/strong&gt;, submitted in March 2023 to ACM Computing Surveys.&lt;/li&gt;
&lt;li&gt;Barranco, F., M. Rodríguez-Álvarez, S. Romero, E. Martínez-Ortigosa. &lt;strong&gt;Learning object contours using bio-inspired spatio-temporal features&lt;/strong&gt;, In preparation, 2023.&lt;/li&gt;
&lt;li&gt;Lobón, F., A. Novo, S. Romero, F. Barranco. &lt;strong&gt;Segmentation of Independently-Moving Objects using neuromorphic sensors&lt;/strong&gt;, In preparation, 2023.
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="authors"&gt;Authors&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Francisco Barranco&lt;/strong&gt; - &lt;em&gt;University of Granada&lt;/em&gt;
Please report problems, bugs, or suggestions to fbarranco_at_ugr_dot_es (Replace &lt;em&gt;at&lt;/em&gt; by @ and &lt;em&gt;dot&lt;/em&gt; by .).&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="acknowledgments"&gt;Acknowledgments&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;This work was supported by the AEI through the grant PID2019-109434RA-I00 / AEI / 10.13039/501100011033.&lt;/li&gt;
&lt;/ul&gt;
&lt;figure&gt;&lt;img src="https://www.ugr.es/~cvrlab/projects/bioperception/aei_logo.jpeg"&gt;
&lt;/figure&gt;
&lt;h2 id="license"&gt;License&lt;/h2&gt;
&lt;p&gt;Copyright (C) 2018 Francisco Barranco, 01/09/2018, University of Granada.&lt;/p&gt;
&lt;p&gt;This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License.&lt;/p&gt;
&lt;p&gt;This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.&lt;/p&gt;
&lt;p&gt;You should have received a copy of the GNU General Public License along with this program. If not, see
.&lt;/p&gt;</description></item><item><title>Drone control based on vision and gestural programming</title><link>https://www.ugr.es/~cvrlab/projects/drone_gesturecontrol/</link><pubDate>Mon, 06 Sep 2021 11:22:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/drone_gesturecontrol/</guid><description/></item><item><title>Drones for video surveillance applications</title><link>https://www.ugr.es/~cvrlab/projects/drone_pursuing/</link><pubDate>Mon, 27 Apr 2020 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/drone_pursuing/</guid><description/></item><item><title>Real-Word anomaly detection in Surveillance through Semi-supervised Federated Active Learning</title><link>https://www.ugr.es/~cvrlab/projects/dl_anomaly/</link><pubDate>Mon, 27 Apr 2020 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/dl_anomaly/</guid><description/></item><item><title>SLAM for unknown scenarios with Cartographer</title><link>https://www.ugr.es/~cvrlab/projects/turtlebot_slamcartographer/</link><pubDate>Mon, 27 Apr 2020 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/turtlebot_slamcartographer/</guid><description/></item><item><title>Egomotion estimation methods for drones</title><link>https://www.ugr.es/~cvrlab/projects/drone_egomotion/</link><pubDate>Sat, 27 Apr 2019 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/drone_egomotion/</guid><description/></item><item><title>Dataset for visual navigation with event-based sensors</title><link>https://www.ugr.es/~cvrlab/projects/neuro_navigationdataset/</link><pubDate>Fri, 27 Apr 2018 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/neuro_navigationdataset/</guid><description>&lt;p&gt;This directory contains Matlab (R) implementations of methods to generate and handle DVS data for building a dataset&lt;/p&gt;
&lt;p&gt;Standardized benchmarks for Computer Vision provide a very powerful tool for the development of new improved techniques in the field. Frame-free event-driven vision still lacks of datasets to assess the accuracy of their methods. Furthermore, the visibility of Event-driven Vision will be enhanced thanks to the comparison to frame-based conventional Computer Vision to make researchers in the field understand the potential of this novel technology.&lt;/p&gt;
&lt;p&gt;Benchmarks for Event-driven Vision and Computer Vision are significantly different. Frame-free sensors collect events that are triggered due to changes in the image while conventional sensors collect the luminance of the scene. If we want to compare methods for heterogeneous sensors, we require a conventional sensor and a frame-free sensor collecting events from the same scene, so that the processing for frame-based and event-driven techniques can be applied to the same data. In our case, we record the sequences with the DAVIS sensor [1] that collects asynchronous events and synchronous frames.&lt;/p&gt;
&lt;h2 id="davis-sensor-calibration"&gt;Davis sensor calibration&lt;/h2&gt;
&lt;p&gt;The DAVIS240b sensor is mounted on a stereo rig with a Microsoft Kinect Sensor that provides the RGB image and the depth map of the scene. The stereo rig is mounted on a Pan Tilt Unit (PTU-46-17P70T by FLIR Motion Control Systems). Finally, the Pan Tilt Unit is on-board a Pioneer 3DX Mobile Robot. The PTU controls the pan and tilt angles and angular velocities, while the Pioneer 3DX Mobile Robot is in control of the direction of translation and the speed. There are ROS (Robot Operating System) packages available for the PTU and the Pioneer 3DX mobile robot. Our dataset provides the following:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The 3D motion parameters: 3D translation of the camera and 3D pose}. They are provided by the PTU and the Pioneer Mobile Robot. Assuming that their coordinate centers are the same, they have to be calibrated with respect to the DAVIS coordinates.&lt;/li&gt;
&lt;li&gt;The image depth. It is obtained by the Microsoft Kinect Sensor (RGB-D sensor). A stereo calibration to register the Kinect depth to the DAVIS camera coordinates is required.&lt;/li&gt;
&lt;li&gt;The 2D motion flow field ground-truth. Using the 3D motion parameters and the depth from the DAVIS coordinate system, the 2D motion flow field ground-truth of the scene is reconstructed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The tasks implemented in this repository are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Calibration of DAVIS and RGB-D sensor&lt;/li&gt;
&lt;li&gt;Calibration of DAVIS and PTU&lt;/li&gt;
&lt;li&gt;Generating 2D motion flow field&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="more-documentation"&gt;More documentation&lt;/h2&gt;
&lt;p&gt;If you use any of the methods or code, please cite the article
F. Barranco, C. Fermuller, Y. Aloimonos, T. Delbruck
A dataset for Visual Navigation with Neuromorphic Methods
Frontiers in Neuroscience: Neuromorphic Engineering, 2015.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;@article{barranco_dataset_2015,
title={A dataset for Visual Navigation with Neuromorphic Methods},
author={Barranco, Francisco and Fermuller, Cornelia and Aloimonos, Yiannis and Delbruck, Tobi},
journal={Frontiers in Neuroscience: Neuromorphic Engineering},
pages={1--16},
year={2015}
}
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Some additional information is available at:
&lt;/p&gt;
&lt;h2 id="example"&gt;Example&lt;/h2&gt;
&lt;p&gt;Please, take a look at reconstructFlow.m for an example that shows in a general way the main functionalities of the repository.
The code uses some data and matfiles that are already available in the repository (./DATA). Anyway, more sequences and ground-truth can be found in the project site. The example file reconstructFlow also includes some comments to understand how the matfiles with the depth and the DAVIS, RGB-D sensor, and PTU calibration parameters have been computed. Note that, all the parameters should be re-computed with a new configuration.&lt;/p&gt;
&lt;h2 id="update-october-2016"&gt;Update: October 2016&lt;/h2&gt;
&lt;p&gt;Please, note that some of the code and the benchmark have been updated.
The values of the flow are computed as m/frame. This means that, since event sequences (.aedat) are recorded at different frame rates, also the time scale changes for each sequence. Take a look at the variable &amp;ldquo;frame_rate&amp;rdquo; (microsecond) in the code to translate into m/microsecond if necessary.&lt;/p&gt;
&lt;p&gt;Please report problems, bugs, or suggestions to
fbarranco_at_ugr_dot_es (Replace &lt;em&gt;at&lt;/em&gt; by @ and &lt;em&gt;dot&lt;/em&gt; by .).&lt;/p&gt;
&lt;p&gt;Copyright (C) 2015 Francisco Barranco, 11/18/2015, University of Granada.&lt;/p&gt;
&lt;p&gt;This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License.&lt;/p&gt;
&lt;p&gt;This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.&lt;/p&gt;
&lt;p&gt;You should have received a copy of the GNU General Public License
along with this program. If not, see
.&lt;/p&gt;</description></item><item><title>Real-time clustering and tracking for event-based sensors</title><link>https://www.ugr.es/~cvrlab/projects/neuro_clusteringiros/</link><pubDate>Fri, 27 Apr 2018 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/neuro_clusteringiros/</guid><description>&lt;p&gt;This package implements a clustering and tracking using Kalman filters for DVS data (event based camera).&lt;/p&gt;
&lt;h2 id="getting-started"&gt;Getting Started&lt;/h2&gt;
&lt;p&gt;Clone or download the project. After that, first source your &lt;em&gt;ROS distro&lt;/em&gt; files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ source /opt/ros/&amp;lt;your_distro&amp;gt;/setup.bash
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;And follow the next files to compile the package:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ mkdir -p ~/dvs_clustering_tracking/src
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ cd ~/dvs_clustering_tracking/
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ catkin_make
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;After this, you will have a copy of the project for testing.&lt;/p&gt;
&lt;h3 id="prerequisites"&gt;Prerequisites&lt;/h3&gt;
&lt;p&gt;In order to install the project, you will need a ROS distribution running in your platform. This project was developed and tested using &lt;em&gt;kinetic&lt;/em&gt;. See more about the installation and tutorial examples in the
.&lt;/p&gt;
&lt;h3 id="installing"&gt;Installing&lt;/h3&gt;
&lt;p&gt;In order to install all the dependencies you will have to install:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;
: where you will require &lt;code&gt;dvs_ros_driver&lt;/code&gt;(and then &lt;code&gt;dvs_driver&lt;/code&gt;), &lt;code&gt;dvs_msgs&lt;/code&gt;, &lt;code&gt;dvs_renderer&lt;/code&gt;, &lt;code&gt;libcaer_catkin&lt;/code&gt; (and then &lt;code&gt;libcaer&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="running-the-tests"&gt;Running the tests&lt;/h2&gt;
&lt;p&gt;After the compiling the project code, source the &lt;em&gt;setup.bash&lt;/em&gt; file&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;:~/dvs_clustering_tracking$ source devel/setup.bash
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Next, we will show two examples for running some of the tests:&lt;/p&gt;
&lt;h3 id="running-the-dvs_renderer"&gt;Running the dvs_renderer&lt;/h3&gt;
&lt;p&gt;If you do not have the DVS camera and want to run a file, simply run:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;:~/dvs_clustering_tracking$ rosbag play ~/bagfiles/shapes_rotation.bag -l
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;where in &lt;code&gt;~/bagfiles&lt;/code&gt; we have the rosbag file with dvs events ([rosbag files dataset] (
))&lt;/p&gt;
&lt;p&gt;Or, if you have your own camera, just run in a different console (you&amp;rsquo;ll need to source again the setup.bash files in this new console):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ roslaunch dvs_renderer dvs_mono.launch
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h3 id="running-dvs_meanshift"&gt;Running dvs_meanshift&lt;/h3&gt;
&lt;p&gt;In case there is no attached DVS, just run the same rosbag example:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;:~/dvs_clustering_tracking$ rosbag play ~/bagfiles/shapes_rotation.bag -l
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;where in &lt;code&gt;~/bagfiles&lt;/code&gt; we have the rosbag file with dvs events ([rosbag files dataset] (
))&lt;/p&gt;
&lt;p&gt;Then, run in a different console (you&amp;rsquo;ll need to source again the &lt;em&gt;setup.bash&lt;/em&gt; files in this new console):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-fallback" data-lang="fallback"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ roslaunch dvs_meanshift dvs_segmentation.launch
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/p-eapPg5BuE?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;h2 id="publications"&gt;Publications&lt;/h2&gt;
&lt;p&gt;If you use this work in an academic context, please cite the following publication:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;F. Barranco, C. Fermuller, E. Ros: &lt;strong&gt;Real-time clsutering for multi-target tracking using event-based sensors&lt;/strong&gt;. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, 2018. (
)&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="authors"&gt;Authors&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Francisco Barranco&lt;/strong&gt; - &lt;em&gt;University of Granada&lt;/em&gt;
Please report problems, bugs, or suggestions to fbarranco_at_ugr_dot_es (Replace &lt;em&gt;at&lt;/em&gt; by @ and &lt;em&gt;dot&lt;/em&gt; by .).&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="acknowledgments"&gt;Acknowledgments&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;This work was supported by a Spanish Juan de la Cierva grant (IJCI-2014-21376), partially funded by MINECO-FEDER TIN2016-81041-R grant, the EU HPB-SGA2 grant (H2020-RIA 785907), the National Science Foundation under grant SMA 1540916, and the NG-UMD seed grant (Object Motion Analysis for Autonomous Systems).&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="license"&gt;License&lt;/h2&gt;
&lt;p&gt;Copyright (C) 2018 Francisco Barranco, 01/09/2018, University of Granada.&lt;/p&gt;
&lt;p&gt;This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License.&lt;/p&gt;
&lt;p&gt;This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.&lt;/p&gt;
&lt;p&gt;You should have received a copy of the GNU General Public License along with this program. If not, see
.&lt;/p&gt;</description></item><item><title>Visuoimitation - Baxter imitates movements looking at an operator</title><link>https://www.ugr.es/~cvrlab/projects/baxter_visuoimitation/</link><pubDate>Fri, 27 Apr 2018 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/baxter_visuoimitation/</guid><description>&lt;p&gt;The objective of the system is to control a baxter robot using the kinect V2 and tis body structure extraction modules (body tracking) under the ROS (Robot Operating System) framework. Modules will be developed to make an interface between the robot and the Kinect body pose estimation so that the robot can imitate the movements of the operator in real time. The system is developed in Ubuntu through the use of libraries usch as libfreenect2, openNI-2, and NITE2 becasue the ROS framework is developed for Ubuntu.&lt;/p&gt;
&lt;h2 id="installation"&gt;Installation&lt;/h2&gt;
&lt;p&gt;The project has been tested with the 14.04 STL Ubuntu distribution. Also, for the current project ROS indigo was installed. In case you need to install this ROS version, you need first:&lt;/p&gt;
&lt;p&gt;Configure the source.list file&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ sudo sh -c 'echo &amp;quot;deb http://packages.ros.org/ros/ubuntu \$ (lsb\_release -sc) main&amp;quot;&amp;gt; /etc/apt/sources.list.d/ros-latest.list'
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Configure the keys&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ sudo apt-key adv --keyserver hkp: //ha.pool.sks-keyservers.net: 80 --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Installation&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ sudo apt-get update sudo apt-get install ros-indigo-desktop-full
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Before using ROS, update rosdep to install all the required dependencies&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ sudo rosdep init
$ rosdep update
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Configure the environment for compilation&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ echo &amp;quot;source /opt/ros/indigo/setup.bash&amp;quot; &amp;gt;&amp;gt; ~/.bashrc
$ source ~/.bashrc
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Installing Gazebo for simulation: First we decided to use catkin&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ mkdir -p ~/catkin_ws/src
$ cd ~/catkin_ws/src
$ catkin_init_workspace
$ cd ~/catkin_ws
$ catkin_make
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Configure the environment&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ echo &amp;quot;source ~/catkin_ws/devel/setup.bash&amp;quot; &amp;gt;&amp;gt; ~/.bashrc
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Install Gazebo&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ sudo apt-get install -y gazebo2
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Donwload all the code for ROS indigo ad build Gazebo&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ cd ~/catkin_ws/src
$ git clone https://github.com/ros-simulation/gazebo_ros_pkgs.git -b indigo-devel
$ cd ~/catkin_ws/
$ catkin_make
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Install the Baxter package to use it with the simulator now&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ cd ~/ros_ws/src
$ wstool init .
$ wstool merge https://raw.githubusercontent.com/RethinkRobotics/
$ baxter_simulator/master/baxter_simulator.rosinstall
$ wstool update
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Compile the package&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ cd ~/ros_ws
$ catkin_make
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Run Baxter for the simulator&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ roslaunch baxter_gazebo baxter_world.launch****
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="workstation-setup"&gt;Workstation Setup&lt;/h2&gt;
&lt;p&gt;The required steps to setup your workstation are described in the tutorial from rethink robotics®. This include the installation of Ubuntu, ROS, the Baxter SDK and the first connection to the robot. The direct connection from the Baxter to your workstation also requires a
.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt; Example: Network Connection-&amp;gt;Add-&amp;gt;Ethernet
Connection name: Baxter
IPv4 Settings: Method: Manual
Address: 169.254.0.10, Netmask: 255.255.0.0, Gateway: 0.0.0.0
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="gazebo-optional"&gt;Gazebo (optional)&lt;/h2&gt;
&lt;p&gt;Gazebo is a powerful simulation tool for robots. It allows to use already existing robots like the Baxter or to create own one. It is very useful for people who have not the possibility to work with a physical robot.
.&lt;/p&gt;
&lt;p&gt;It is more comfortable to have a separate workspace for the physical robot and the visual one in the simulation. It is for example not necessary to change everytime the baxter.sh file. The tutorial from the
explains the steps and it is a good way to get in touch with gazebo.&lt;/p&gt;
&lt;p&gt;If MoveIt is already installed the packages must be copied from the workspace ros_ws in the src folder from ros_gazebo_master. But to use the simulation from this website don&amp;rsquo;t modify the following line from baxter_world.launch.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt; &amp;lt;arg name=&amp;quot;world_name&amp;quot; value=&amp;quot;$(find baxter_gazebo)/worlds/baxter.world&amp;quot;/&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="robot-arm-calibration"&gt;Robot arm calibration&lt;/h2&gt;
&lt;p&gt;The calibration of the arms is recommended to get precisely results. There are two different calibration routines which are very well described on the website from rethink robotics.
.&lt;/p&gt;
&lt;h2 id="examples-and-code"&gt;Examples and code&lt;/h2&gt;
&lt;p&gt;The main objective of the project is described in the initial section of this page. The following videos show first and example of the setup with the code running, and the Baxter simulation in Gazebo imitating the movements of the operator. The next video shows a successful example of the robot imitating the operator movements.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/KZcqivsb1dM?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The next video shows a successful example of the robot imitating the operator movements.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/V6VbvJ-t0vg?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;h2 id="contact"&gt;Contact&lt;/h2&gt;
&lt;p&gt;Granada
Cristina Garrido
&lt;/p&gt;</description></item><item><title>Object manipulation with the Baxter robot</title><link>https://www.ugr.es/~cvrlab/projects/baxter_objectmanipulation/</link><pubDate>Thu, 27 Apr 2017 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/baxter_objectmanipulation/</guid><description>&lt;p&gt;This website presents various demonstrations of pick and place tasks with the Baxter research robot. It contains object recognition and manipulation with different focus areas such as various shapes, sizes, and colours. It also includes trajectory and motion planning and the use of different grippers with the physical robot and in the simulation with gazebo, too. The demonstrations, based upon a project from Abhishek Patil from the Northwestern University. (
)&lt;/p&gt;
&lt;h3 id="contents"&gt;Contents&lt;/h3&gt;
&lt;p&gt;The packages include the following files:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;baxter_pnp.py&lt;/strong&gt;&lt;br /&gt;
It includes the commands for the pick and place task and subscribes to the topic &amp;ldquo;detected objects&amp;rdquo; which contains the coordinates of the objects. The MoveGroupCommander and the PlanningSceneInterface are used for the movements.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;baxter_img.cpp&lt;/strong&gt;&lt;br /&gt;
The file is for the object detection with the Baxter hand camera. The tool OpenCV is used and the results are published to the topic &amp;ldquo;detected_objects&amp;rdquo;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;baxter_pick_and_place.launch&lt;/strong&gt;&lt;br /&gt;
The launch file runs all the required nodes for the specific project.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Folder with models for simulation&lt;/strong&gt;&lt;br /&gt;
The projects for the simulation contain some models for the objects and the table which are from the Rethink Robotics example.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
&lt;h2 id="workstation-setup"&gt;Workstation Setup&lt;/h2&gt;
&lt;p&gt;To set the workstation, the tutorial from
, in this site you can find the installation of Ubuntu, the ROS specific version for that Ubuntu version, the Baxter SDK and the instructions for establishing the first comunication with the robot, which requires an Ethernet connection.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;Example: Network Connection-&amp;gt;Add-&amp;gt;Ethernet
Connection name: Baxter
IPv4 Settings: Method: Manual
Address: 169.254.0.10, Netmask: 255.255.0.0, Gateway: 0.0.0.0
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="moveit"&gt;MoveIt&lt;/h2&gt;
&lt;p&gt;
is used for the motion planning, manipulation and collision checking.&lt;/p&gt;
&lt;p&gt;Moreover the package
by Michael Ferguson needs to be installed.&lt;/p&gt;
&lt;h2 id="robot-arm-calibration"&gt;Robot arm calibration&lt;/h2&gt;
&lt;p&gt;It is recommended to calibrate Baxter&amp;rsquo;s arms for getting more precise results.
there are two different calibration routines which are very well described.&lt;/p&gt;
&lt;h2 id="table-position-and-zero-point-in-moveit"&gt;Table position and zero point in MoveIt&lt;/h2&gt;
&lt;p&gt;It is very annoying to find every time you change the position of the robot or the used table to get the right sizes and coordinates manually for the simulation in MoveIt!. There are two programs which make this task easier. The first one is used to get the distance from the ground to the zero point from the reference frame &amp;ldquo;base&amp;rdquo; from MoveIt!. This value is always the same if the robot position doesn&amp;rsquo;t change. The table height must be smaller than 0.9 m and known to get this distance.
&lt;/p&gt;
&lt;p&gt;Then you can continue with the second program which finds the size and the position of the table. At first add the measured distance from the task before.
&lt;/p&gt;
&lt;h2 id="the-color-of-the-objects"&gt;The color of the objects&lt;/h2&gt;
&lt;p&gt;The object recognition can only work if the colors of the objects are known. The colors are defined in the baxter_img.cpp file as low and high hsv values. The example colored_papers.pdf includes three different colors with the related hsv values. They can be printed and used for a first test to recognize them. Download the colored papers! But this aren&amp;rsquo;t the exact values because it depends on the printer settings, the ink, the paper, etc. The best way to get the color values from an object which should be detected is to take pictures with the Baxter camera from it on the workspace in different positions. This can be in the shadow from something or at a highly illuminated area. Moreover, it is recommended to have every time all the used colors on every picture to see that only the color with the specific hsv values is detected.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ rosrun image_view image_view image:=/cameras/right_hand_camera/image
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This command opens a window with a stream of images from Baxter&amp;rsquo;s right hand camera and allows to save images by right-clicking on the display window.
Then the images can be analysed with the
. Open a new terminal and execute the python file from the location where it is saved.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ python hsvThresholder.py frame0000.jpg
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="camera-calibration-and-other-parameters"&gt;Camera calibration and other parameters&lt;/h2&gt;
&lt;p&gt;The camera calibration is always necessary because every camera is different. The parameters can be determined with the
. The website explains the calibration and optimization from a camera with a printed checkerboard. The first calibration example, can be done with an example based on a total of 20 images to learn the process. Another option without the software Matlab is the tutorial
. The parameters must be changed in the baxter_img.cpp file which is in the package from the projects.&lt;/p&gt;
&lt;p&gt;The distance from the camera to the object must be changed in function &amp;ldquo;Get3DPos&amp;rdquo; which is in the same file. (475 mm) Moreover must be the offsets &amp;ldquo;0.07+0.57&amp;rdquo; and &amp;ldquo;0.032&amp;rdquo; checked. They are necessary to get the coordinates in the Baxter coordinate system.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;pose.position.x = rel_pos[i].x / 1000+0.07+0.57;
pose.position.y = rel_pos[i].y / 1000-0.032;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;One object can be placed on the table and the position can be determined by the gripper position.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;rostopic echo /robot/limb/left/endpoint_state/pose -n 1
rostopic echo detected_objects -n 1
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then must be the detected position of this object compared with the real one. The differences are the offsets.
The factors 0.015 and 0.058 correct the small deviations from the functions
(int) rect.center.x- WIDTH / 2 and -((int) rect.center.y- HEIGHT / 2) which are used in the code.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;float tempx = _Ma(0, 0, INVTRANS) * src-&amp;gt;x + 0.015;
float tempy = _Ma(1, 1, INVTRANS) * src-&amp;gt;y + 0.058;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The following example shows the equations for the camera resolution 1280x800.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;2*(1280/2-c_x)/f_x and 2*(800/2-c_y)/f_y
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Further changes for the size of the objects are explained in the comments of the code.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Change grippers&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;At first the arguments in the demo_baxter.launch must be changed to true.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt; &amp;lt;arg name=&amp;quot;load_robot_description&amp;quot; default=&amp;quot;true&amp;quot;/&amp;gt;
&amp;lt;arg name=&amp;quot;right_electric_gripper&amp;quot; default=&amp;quot;true&amp;quot;/&amp;gt;
&amp;lt;arg name=&amp;quot;left_electric_gripper&amp;quot; default=&amp;quot;true&amp;quot;/&amp;gt;
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Then can the grippers be changed in the files left_end_effector.urdf.xacro and right_end_effector.urdf.xacro.
The possible settings for this files are explained at
.&lt;/p&gt;
&lt;h2 id="tasks-overview"&gt;Tasks Overview&lt;/h2&gt;
&lt;p&gt;The tasks use two different path planning functions which have advantages and disadvantages. The movements with the common function are in the most cases faster and smoother but sometimes it computes very strength pathes. The cartesian path planning allways plans straight ways to the added waypoints. This can be very important for tasks, like the pick and place tasks with the basket, where the orientation should be the same. The disadvantages are that the function can&amp;rsquo;t allways find a path to the goal and that it doesn&amp;rsquo;t wait to continue with the code until it finished. (See Troubleshooting section) Moreover the tasks present various variations to solve a task with different performances.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/nwvRZnX8Rho?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;Measure MoveIt zero point&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;This package is used to get the distance from the ground to the zero point from the reference frame &amp;ldquo;base&amp;rdquo; from MoveIt. This value is needed for the collision checking and is allways the same if the robot position doesn&amp;rsquo;t change. The table height must be smaller then 0.9 m and added to the variable table_size_z. The right arm move from the start position down and measure with the infrared sensor the distance to the table and calculate the zero point with the current position, the distance to the table and the table height. The result will be printed on the screen and is required for all tasks.&lt;/p&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is build from a sourced terminal.
Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch baxter_measure_zero_point baxter_pick_and_place.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Build a tower with one arm&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;In this project the Baxter build the tower only with the right arm. The right hand camera detect every time at the start position the objects on the table and they will be sorted by size. Then it allways pick the biggest one and build a tower. It approach the object and orientate the gripper to the right angle and if it failed to pick the object it try it again. This continue until the table is clear. The program require the values for the variable offset_zero_point, the table size and position.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/fi68d5B1JOE?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;Build a tower with one arm and detect the objects only one time&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;This change improves the performance of the Baxter because it detects the objects only at the start and at the end to look if the table is clear.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/KQ7efHl967M?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is build from a sourced terminal.
Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch baxter_pnp_one_arm baxter_pick_and_place.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Build a tower with one arm in Gazebo&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;This is the same task in the simulation with Gazebo but the camera isn&amp;rsquo;t used and the position of the objects are fixed.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/wKhNArqU0MA?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The project must be added in the src folder from the workspace for the simulation and can be launched after it is build from a sourced terminal.
Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ source ros_gazebo_master/devel/setup.bash
$ cd ros_gazebo_master
$ ./baxter.sh sim
$ catkin_make
$ roslaunch baxter_gazebo baxter_world.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Open a second new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ source ros_gazebo_master/devel/setup.bash
$ cd ros_gazebo_master
$ ./baxter.sh sim
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;em&gt;Wait until the message &amp;ldquo;Simulator is loaded and started successfully&amp;rdquo; in the first terminal appears.&lt;/em&gt;&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ rosrun baxter_tools enable_robot.py -e
$ roslaunch baxter_pnp_one_arm_cartesian_sim baxter_pick_and_place.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Build a tower with both arms alternately&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;In this project the Baxter build the tower with both arms alternately. Both of the hand cameras detect every time at the start position the objects on the table and they will be sorted by size. The arms pick only the objects on their side and allways start with the biggest one and build a tower together. The grippers approach the objects and orientate to the right angle and if they fail to pick an object they try it again. This continue until the table is clear. The program require the values for the variable offset_zero_point, the table size and position.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/6UzWeti0UB4?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is build from a sourced terminal.
Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch baxter_pnp_both_arms_alternately baxter_pick_and_place.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Pick and Place with a basket&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The task from Baxter is to grip a pink basket with the left gripper and throw the cubes with the right gripper in it. It starts to find the position of the basket. At first the right hand camera detect the it from the start position and move to it. Then it looks a second time because if the basket is not directly under the camera it will also detect the sides from the basket and this cause deviations.&lt;/p&gt;
&lt;figure&gt;&lt;img src="https://www.ugr.es/~cvrlab/projects/baxter_objectmanipulation/basketview.jpg"&gt;
&lt;/figure&gt;
&lt;p&gt;In the next step moves the right arm to the side of the table and the left arm approach the detected position with a specific orientation. It moves a little bit further than the pick goal to have it very close in the gripper. Then the left gripper grip it and moves to the start position for the left arm which has specific joint coordinates because they allows a high probability that a valid cartesian path can be found. The right hand camera detect every time at the start position the objects on the table. The right gripper approach the object and orientate to the right angle and pick the object. If it failed to pick an object it try it again. Then the right gripper moves up and the left gripper moves with the basket under it. The cube will be thrown in the basket and this continue until the table is clear. The program require the values for the variable offset_zero_point, the table size and position.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/cJtrwRFSiCc?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is build from a sourced terminal.
Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/indigo/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch baxter_pnp_basket_cartesian baxter_pick_and_place.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Troubleshooting&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The projects include some features to solve problems when the robot failed. The following video will explain them.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/0TOK8PX-DxI?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The simulation in Gazebo has sometimes the problems that the object slip out of the gripper or it doesn&amp;rsquo;t stop to move.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/IT4SCURQNSk?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;&lt;em&gt;Errors&lt;/em&gt;
“Invalid Trajectory: start point deviates from current robot state more than …“&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ rosparam set /move_group/trajectory_execution/allowed_start_tolerance 0.0
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;&lt;strong&gt;Copyright (C) 2017 Maik Heufekes, 05/07/2017.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License.&lt;/p&gt;
&lt;p&gt;This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.&lt;/p&gt;
&lt;p&gt;You should have received a copy of the GNU Lesser General Public License along with this program. If not, see
.&lt;/p&gt;</description></item><item><title>Contour detection and proto-segmentation with event sensors</title><link>https://www.ugr.es/~cvrlab/projects/neuro_iccv16/</link><pubDate>Wed, 27 Apr 2016 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/neuro_iccv16/</guid><description/></item><item><title>Object classification with the Baxter robot</title><link>https://www.ugr.es/~cvrlab/projects/baxter_conveyor/</link><pubDate>Wed, 27 Apr 2016 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/baxter_conveyor/</guid><description>&lt;p&gt;This project shows the development of various classification tasks in static and dynamic environments with the Baxter robot. This robot will separate two groups of objects with different colours, positions and sizes, using the Robot Operating System (ROS Kinetic) on Ubuntu 16.04 LTS, MoveIt! and the OpenCV library for the computer vision, based on the colour filter made by
. Check the
.&lt;/p&gt;
&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
&lt;h2 id="workstation-setup"&gt;Workstation Setup&lt;/h2&gt;
&lt;p&gt;To set the workstation, the tutorial from
, in this site you can find the installation of Ubuntu, the ROS specific version for that Ubuntu version, the Baxter SDK and the instructions for establishing the first comunication with the robot, which requires an Ethernet connection.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;Example: Network Connection-&amp;gt;Add-&amp;gt;Ethernet
Connection name: Baxter
IPv4 Settings: Method: Manual
Address: 169.254.0.10, Netmask: 255.255.0.0, Gateway: 0.0.0.0
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="moveit"&gt;MoveIt&lt;/h2&gt;
&lt;p&gt;
is used for the motion planning, manipulation and collision checking.&lt;/p&gt;
&lt;p&gt;Moreover the package
by Michael Ferguson needs to be installed.&lt;/p&gt;
&lt;h2 id="robot-arm-calibration"&gt;Robot arm calibration&lt;/h2&gt;
&lt;p&gt;It is recommended to calibrate Baxter&amp;rsquo;s arms for getting more precise results.
there are two different calibration routines which are very well described.&lt;/p&gt;
&lt;h2 id="colour-of-the-objects"&gt;Colour of the objects&lt;/h2&gt;
&lt;p&gt;The object recognition can only work if the colours of the objects are known. The colours are defined in the baxter_img.cpp file as low and high HSV values. The best way to get the color values from an object which should be detected is to take pictures with the Baxter camera from it on the workspace in different positions and with different illuminations.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ rosrun image_view image_view image:=/cameras/right_hand_camera/image
&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This command opens a window with a stream of images from Baxter&amp;rsquo;s right hand camera and allows to save images by right-clicking on the display window.
Then the images can be analyzed with the
. Open a new terminal and execute the python file from the location where it is saved.&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ python hsvThresholder.py frame0000.jpg
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="tasks-overview"&gt;Tasks Overview&lt;/h2&gt;
&lt;p&gt;There are two main classification tasks using a paddle and a wedge as tools, the first one is used in static classification and the other one in dynamic. Both of them are made out of cardboard and it is common that sometimes the objects weight more than the tools, resulting on difficulties at the time of classifying. The clustering algorithm used is K-means with k = 2, it separates the objects by colour and position. This algorithm gives us the centroids of both groups so that we can get the middle point.&lt;/p&gt;
&lt;h3 id="static-environment-classification"&gt;Static environment classification&lt;/h3&gt;
&lt;p&gt;Baxter classifies objects on the top of a table, setting them aside.&lt;/p&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is built from a sourced terminal.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/wTfLTKC_PFo?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/kinetic/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch classif classification.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;h3 id="dynamic-environment-classification-using-a-wedge"&gt;Dynamic environment classification using a wedge&lt;/h3&gt;
&lt;p&gt;For making this classification, the speed of the objects on the conveyor belt has to be calculated. Then, the Baxter robot will predict where the objects are to separate them.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/crF7NCy9WJw?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is built from a sourced terminal.
Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/kinetic/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch dyn_wedge dyn_wedge.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;h3 id="dynamic-environment-classification-using-a-paddle"&gt;Dynamic environment classification using a paddle&lt;/h3&gt;
&lt;p&gt;For making this classification, the speed of the objects on the conveyor belt has to be calculated. Then, the Baxter robot will predict where the objects are to separate them. In this case, the tool used is a paddle, so it will be necessary to predict the position twice, as it can be seen in the video.&lt;/p&gt;
&lt;p&gt;The project must be added in the src folder from the workspace and can be launched after it is built from a sourced terminal.&lt;/p&gt;
&lt;div style="position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden;"&gt;
&lt;iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" loading="eager" referrerpolicy="strict-origin-when-cross-origin" src="https://www.youtube.com/embed/BypfZhM5Lto?autoplay=0&amp;amp;controls=1&amp;amp;end=0&amp;amp;loop=0&amp;amp;mute=0&amp;amp;start=0" style="position: absolute; top: 0; left: 0; width: 100%; height: 100%; border:0;" title="YouTube video"&gt;&lt;/iframe&gt;
&lt;/div&gt;
&lt;p&gt;Open a new terminal:&lt;/p&gt;
&lt;pre&gt;&lt;code&gt;$ source /opt/ros/kinetic/setup.bash
$ source ros_ws/devel/setup.bash
$ cd ros_ws/
$ ./baxter.sh
$ catkin_make
$ roslaunch dyn_classif dyn_classif.launch
&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id="packages"&gt;Packages&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;classif&lt;/strong&gt;. It includes the classification in a static environment.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;dyn_classif&lt;/strong&gt;. Classification in a dynamic environment using a paddle.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;dyn_wedge&lt;/strong&gt;. Classification in a dynamic environment using a wedge.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;pick_and_place&lt;/strong&gt;. Adapted pick and place program for executing it in the Baxter robot.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The files included by this packages are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;baxter_img.cpp: This C++ program identifies the objects in the image, visualizes them and calculates the middle point between the two groups of objects, sending them to the control file.&lt;/li&gt;
&lt;li&gt;file.py: It is the control file, it subscribes to the &amp;ldquo;clasificacion&amp;rdquo; or &amp;ldquo;detected_objects&amp;rdquo; topic and plans the trajectories and execute them to separate or pick the objects.&lt;/li&gt;
&lt;li&gt;file.launch: This launch file runs all the required nodes to accomplish the task.&lt;/li&gt;
&lt;/ul&gt;</description></item><item><title>FPGA implementation of bottom-up attention with top-down modulation</title><link>https://www.ugr.es/~cvrlab/projects/fpga_attention/</link><pubDate>Mon, 27 Apr 2015 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/fpga_attention/</guid><description/></item><item><title>FPGA implementation of optical flow, disparity, and low-level features</title><link>https://www.ugr.es/~cvrlab/projects/fpga_ofdisp/</link><pubDate>Mon, 27 Apr 2015 00:00:00 +0000</pubDate><guid>https://www.ugr.es/~cvrlab/projects/fpga_ofdisp/</guid><description/></item></channel></rss>