# data[].attributes.attachment_link.title
GitHub - Adlink-ROS/ros_menu: Neuron Startup Menu for ROS environment setting

# data[].attributes.attachment_link.description
Setting ROS environment every time is an annoying job. ADLINK provides Neuron Startup Menu to make your life easier. Neuron Startup Menu offers these features: Install ROS 1, ROS2, and Cyclone DDS for you automatically. Prompt ROS menu whenever you opening a new terminal.

# data[].attributes.attachment_link.readable_text
最近有几个人加我，都说是刚开始学激光slam，基本都是研一，也有一些大四的（大四的都开始学SLAM了！）。

情况也都差不多，有的是课题组里只有自己做slam，有的是完全没人带，有的是完全没基础，c++都没学过的那种。

这些小白都不知道怎么进行学习，问了我好多相同的问题，所以我就将我自己的思路整理了一下，也就是这篇文章的由来．

第一部分是我自己的SLAM接触史，第二部分是为硕士小白入门写的推荐学习路线，第三部分是为了SLAM爱好者入门写的推荐学习路线．

长文，无图，做好准备．以下内容仅为我自己的观点，有可能有错误及不足，望见谅．

1 我的SLAM接触史

1.1 c++

我也是研一的时候知道自己课题是SLAM相关的。

所以就在2016年10月份的时候开始看 c++ prime plus。之前我没接触过编程，就大一时候上过c语言的课，学完了也没记得什么，之后也没再碰过代码。

这时候还不知道咋编代码，咋运行程序，所以 c++ prime plus 就是光看，没去实际编代码。

1.2 ubuntu

到了2017年3月，课题正式确定，《基于SLAM的室内移动机器人导航技术研究》。

经过同学的帮助，帮我装了人生第一个双系统，ubuntu1404，开始正式接触linux系统。

1.3 ROS

之后边学ubuntu的命令行（ 简单教程），边去学ROS ，ROS的wiki地址为 wiki.ros.org 。

ROS的wiki上的知识实在是太多了，有太多太多的网页跳转，导致我花了太多的时间用在逛wiki上，当时存了30多个书签等着我一个一个的去看。

其实只要学完 Beginner Level 的教程就可以了，这个之后详细说明。

1.4 ROS By Example 1

之后去看了这本书《ros by example 1》，这本书对我来说真的非常重要，让我跑通了第一个ros的demo，十分有成就感。

这本书有配套的代码，可以直接运行，照着书中的步骤一步一步运行即可。学了这么久，通过这本书里的例子，终于在rviz中看到了turtlebot，并让机器人成功的动起来了，真是非常有成就感的一件事。

1.4 概率机器人

忘记了看这本书的具体时间了，只记得实验室里有这本书，然后我实习的时候借去看了一遍。这本书真的是很厉害，里边有关于机器人定位与建图的所有概念和公式。

强烈建议入门的时候不要读这本书，不要读，不要读 。

因为当时我看完了一遍，真是一点都没理解，也一点没记住。这本书就是现在的我来看，也有很多不理解，读不懂的地方，更别说刚入门的小白了。

1.5 小强机器人

之后，我去公司实习，实习的时候有个机器人可以让我玩，小强机器人。这是第一次弄机器人，也是第一次在机器人上操作ROS。

实践ROS与操作机器人大概花费了我至少2个月的时间。时间很长，因为可能突然出现有些问题导致你卡住，进行不下去了。

记得很清楚，在笔记本上rviz点击下发move_base的目标点，机器人始终不动，这个问题卡了我2周的时间。原因是没有在机器人里的/etc/hosts里添加笔记本的ip。。。

所以，如果你课题组里只有自己做SLAM或者机器人相关，一定要多加几个 社群 ，qq群也好，微信群也好，一定不要自己一个人去研究．有很多小问题在别人那只是一句话就能解决的事，自己研究可能就要耽误2周甚至更长的时间。而且，现在做社群的有很多，搜索也很容易找到。

1.6 cartographer

再之后，17年12月份开始，到18年1月份，弄了将近1个月如何使用carographer。弄懂如何跑demo之后，又开始学如何修改launch文件，如何写urdf。

再之后，就是做导航相关的了。

回顾这一年的时间，由于没人带真是走了挺多弯路，学习也没有抓住重点。当然，学过的所有知识都不是无用的，只是没到需要用的时间而已。同时，由于之前基础打的牢，学的多，所以现在我对ROS相关的理解非常深刻，操作也很流畅。

接下来，我将我认为的入门需要学的最重要的东西以及流程写下来，希望可以帮助小白们知道如何学习SLAM，以及如何做SLAM相关的科研。

2 硕士小白做激光SLAM相关课题的学习路线

硕士同学如果课题是SLAM相关的，该如何尽快入门呢？课题组只有自己一个做SLAM的该怎么办呢？

不要慌。目前大部分做SLAM的几乎都是自学的，即使是课题组有其他做SLAM的也只能是让自学的速度更快一点而已，因为知识始终是要自己理解了之后才能成为自己的。多加几个 社群 多交流，也是一样的．

2.1 第一步：双系统

我认为的第一步就是先要装个 双系统 ，windows+ubuntu ，不建议使用虚拟机，虚拟机本身会有很多问题，小白根本不知道出现的问题是虚拟机本身的问题还是环境没弄好导致的问题。相信每个做SLAM的都自己装过不下1遍的双系统。

因为ubuntu系统不是十分健壮，有时候装环境，装驱动尤其是显卡驱动，很可能就把系统弄坏了。有时候强制关机也会导致进不去系统。

由于每个人的电脑配置不一样，导致装双系统的时候会出现各种小问题，花时间都是可以解决的。

装双系统这一阶段大概花费 1天到2周 不等的时间。

2.2 第二步：ubuntu的基础指令

这一步很简单，只要稍微学习下ubuntu的最基础的指令就可以了，可以照着这个 简单教程 进行学习，也可以在b站中找视频进行学习。

一定要记住，刚开始只需要学习最基础的指令就够了，如 如何打开终端，如何复制移动文件，如何新建文件夹等等。

不要学的太多，像shell编程，脚本编写执行等等。很多知识是会在以后的使用中，边学边查就够了，不要第一时间全都去学，基础不够的情况学了再多也不一定能理解和记住。

这一阶段会花费 1天到1周 不等的时间。

2.3 第三步：学习ROS

2.3.1 为什么要学ROS

ROS最大的贡献就是为机器人相关的开发者，统一了接口与操作方式。让开发者可以聚焦于上层的算法设计，不用每个人都去重复的去做通讯，驱动等底层的搭建。同时，ros中也提供了能让各种传感器数据可视化的软件Rviz，以及查看各个节点的关系可视化图等特别好用的工具。

而且，目前的激光SLAM，不管是二维的还是三维的，几乎都是使用ROS的接口编写的，不会用ROS你就不会启动SLAM。所以，ROS是必须要学的。

2.3.2 安装ROS

安装ROS网上有太多的教程，这里就不再说明了，一般照着ROS wiki的 安装教程 一步一步安装都会成功的。

2.3.3 怎么学ROS

ROS的wiki网站( ros.wiki.org)里有几乎所有关于ROS的教程与信息，在学习ROS的时候，最好直接看wiki上的内容，不要去找一些其他网站翻译之后的内容，这里的英文很简单，一般都能直接看懂。

能找到的翻译过的文章大都只是简单的进行翻译与复制，不一定准确不说，往往都是只翻译一个章节或者一小部分，会让人学的不够全面与透彻。

2.3.3.1 ROS Tutorials Beginner Level

首先要照着ROS的官方教学内容进行学习，也就是 ROS Tutorials 的 Beginner Level 内容进行学习，一定要保证每个章节都弄懂，保证每个章节的内容都自己跟着实践过不止1次。只有自己实际操作之后，才能真正的理解，光看是看不会的！
这20个章节真的非常重要，有的时候我也会去再重温一下这20个章节的内容，这真的是使用ROS的最核心的内容，一定要好好学习，要真正的学会，理解。
这一阶段大概花费 2周到3周 的时间。

2.3.3.2 launch文件

学完了初级教程之后，先不要去学中级教程以及其他的教程，因为其他的教程是短时间内用不到的。ROS Tutorials中有2个常用的知识点没有说，一个是用于启动多个节点的roslaunch命令与launch文件，另一个是URDF模型。

launch文件的语法可以去wiki上学一下 http:// wiki.ros.org/roslaunch .

wiki上的内容很多，可以不用全部学，要学的只有2个，一个是如果使用roslaunch命令启动launch文件，命令的格式如下所示

$ roslaunch package_name file.launch

另一个是launch文件的XML语法 。

launch文件的具体教学可以在百度里搜一搜，也可以在任意一个ros的包中查看launch文件的具体内容。

这一阶段最多花费 1天 的时间。

2.3.3.3 URDF

URDF是用来描述传感器坐标系与机器人坐标系间的物理关系的，也就是坐标变换。

这个文件可能第一次看感觉很复杂，又是一种新的语言格式，但是其内容模式是固定的，只需要大段的复制，简单的更改其中的坐标系名字与对应的坐标变换的具体数值，即可完成自己机器人的URDF模型的搭建。

如果没看明白完全没有关系，因为大部分SLAM算法已经编写好了对应的URDF模型，你理解了URDF的概念也是完全可以的，以后再来弄清楚。

这其中可能涉及到一些TF的知识，可以暂时不用管他，只要知道这是ROS帮助我们管理传感器坐标系与机器人坐标系间坐标变换关系的即可。TF也是十分必要的，是之后的必学项

这一阶段最多花费 1天 的时间。

2.4 第四步：试着跑通现有的SLAM算法

如果一切顺利的话，学到这差不多花费一个月的时间，如果你学的快的甚至不需要一个月的时间。

下一阶段为尝试在ROS下跑通现有的SLAM算法。

如果你是做二维的激光SLAM，建议优先跑通 gmapping ，因为这个SLAM算法网上能找到的教程非常多。对应的数据bag文件（bag会在 ROS Tutorials 中学到）可以在 http:// download.ros.org/data/g mapping/ 中获取。

如果是做三维的激光SLAM，建议优先跑通 LOAM ，这个SLAM的数据包及如何运行的教程网上也是非常多。

再跑demo的过程中，你会体验到如何运行launch文件，如何在launch文件中修改订阅的话题的名称，以及如何播放bag文件。同时，也要体会激光雷达的数据是什么样的，构建出的二维地图或者三维地图是什么样的。

这一阶段大概花费 1周到3周 的时间。

2.5 第五步：理解SLAM各个模块的理论概念以及调研SLAM

经过上面的步骤，相信你大致知道SLAM是干什么的了，就是为了构建地图用的，这个地图可以保存下来，用于后续的定位及导航避障中。也有一些SLAM作为里程计在使用，始终提供估计的位姿．

现在，我们可以更深入的学习下SLAM的概念了。我是觉得，只有先实际跑了下SLAM，再去看理论，才能理解SLAM到底是要干什么．要不然直接学习SLAM的各个理论概念，可能等你学完了也没有理解，过一阵时间就忘了．

目前，主流SLAM的结构基本都差不过，分为前端里程计，后端优化，回环检测三个大模块．前端的目的是始终累加位姿，作为里程计使用；后端使用图的结构模型，优化整体位姿，减小前端里程计产生的累计误差；回环检测可以提供一个更强烈的图结构的约束，能够更好的减小累计误差．

这一阶段就是理解SLAM的概念，可以在网上搜下相关的文章，也可以去看下论文，如中文的博士论文，国外的论文；推荐优先看博士论文的综述章节，以及SLAM综述相关的文章，这可以让你很快的理解SLAM各个模块的概念．
调研主流SLAM算法
理解了SLAM概念的时候，相信综述里会列举很多SLAM算法，可以顺便调研下目前的主流SLAM算法方案有哪些，每个SLAM算法的框架是什么样的．

一些传统的SLAM算法如下所示．

二维激光SLAM：hector，gmapping，karto，cartographer等等．

三维激光SLAM：LOAM，Lego-LOAM，LIO-SAM，HDL graph SLAM，cartographer等等．

还有些语义的SLAM我不太懂这里就不说了．

这一阶段大概花费 1周到2周 的时间。

2.6 第六步：阅读文献找创新点

完成以上步骤大概已经过去至少2个月了，相信这时的你已经对SLAM有了一定的理解．接下来就是要为了做课题而改进SLAM．

既然是找创新点，首先就要了解最新SLAM论文的改进点，也就是要疯狂看论文了．我是觉得，只有先理解了SLAM各个模块的理论概念，再看论文才能真正理解论文说的是啥，才能理解论文的创新点在哪．

看论文是为了找创新点，同时也要选择一个现有的SLAM框架，因为从零搭一个SLAM算法是非常难的，所以只能在现有SLAM框架上做改进．

这一阶段可能花费 1个月到2个月 的时间。

2.7 第七步：跑通选定的SLAM框架

将选定的SLAM算法跑通．

如果没有硬件的同学可以直接下载网上的数据集，将数据集和选定的SLAM算法调通．

如果是有机器人以及激光雷达的同学这一步还要多花费一些时间将选定的SLAM算法与自己的硬件及机器人调通，做好心里准备，这一步可能会出现各种问题导致你卡住．

针对自己的硬件，首先要学会如何启动雷达，如何配置雷达；如果是要做移动机器人的，还要学习一下移动机器人如何操作，也就是如何用ROS操作机器人；再之后是如何将自己的硬件与SLAM算法调通，这步网上的教程很多，一定要注意下话题的名字对不对的上．

跑通选定的SLAM算法的意思是 将数据集或者实际的传感器接入到SLAM中，并启动成功，以及保存地图．之后，要再试着改一改这个SLAM的参数，看看不同的参数更改了会带来什么影响．

这一阶段可能花费 1周到2个月 的时间。

2.8 第八步：学习C++

如果你已经找到了想要做的创新点，接下来的步骤就是要将想法通过代码实现出来．

之前的步骤并没有涉及到编程语言，因为最开始真的是用不到自己编写代码．由于ROS社区的贡献，导致机器人开发人员的编写代码量骤降，能想到的功能基本开源社区里都有，只需要跑通调参就可以了．

现在要自己实现新功能，需要自己写代码，所以这一阶段才开始学习编程语言．

语言推荐C++，因为大部分SLAM是用c++编写的．优先推荐阅读 c++ prime plus ，也可以在网上找一些其他的教程．

不管通过什么方式学习 C++ ，一定多多进行编程练习，学习编程只通过看书是永远也学不会的，只有自己多编，多练，才能让自己熟练准确的编程．

这一阶段可能花费 1个月 的时间，也可以在之前的步骤当中抽空学一下。

2.9 第九步：阅读SLAM源码

学完了C++，并不意味着你能够编写SLAM以及自己的想法了．因为编写SLAM是一件挺工程，挺需要经验的事情．

这时的你，可能会陷入到想要实现某些功能，但是完全不知道该怎么写代码的阶段．

这时候就需要你去阅读想要改进的SLAM的代码了．看代码的目的是为了学习如何将实现自己想法实现，以及自己的功能添加到SLAM的哪里．可以只看自己想要改进模块的代码，也可以将整个SLAM的代码全看了，只要知道了自己的功能如何实现了就可以．
C++ 11
由于一般的SLAM的实现都用到了很多C++ 11的新特性，所以看代码的时候也可以简单学一学 C++ 11 相关的知识，以遍更好的阅读代码．
参数
看代码的时候要将每个参数在哪里使用的，这个参数具体代表什么意思弄懂，这样可以让你更好更准确的调参．

同时，也要学习下SLAM的具体实现该怎么写代码，思考自己想要实现的功能该如何写代码，该添加到SLAM的哪部分代码中，要带着问题去读代码．

这一阶段可能花费 1个月到2个月 的时间。

2.10 第十步：编写代码，验证想法

知道了代码怎么写，在SLAM的哪个部分添加，就可以上手写代码了．

新手写代码，调试代码，这一阶段的时间可能花费的要久一点．

写完了代码之后，就是要通过与之前的SLAM算法对比轨迹精度，地图精度等等．输出图片，表格，曲线，证明你的算法更加优秀．

再之后就是写论文，发论文，写硕士论文，毕业．

有些学校要求 工作量 ，这一点一定提前考虑好．SLAM相关功能的改进，撑死了一个改进点一个大章节，编程是不算工作量的，能算在论文里的工作只有改进之后的对比图，对比曲线．

一篇硕士论文大概4-5个大章节，其他章节的内容怎么写，一定要提前考虑好．不要编程编了2年，最后写论文发现只能算1个大章节，那就尴尬了．

3 单纯的喜欢激光SLAM，非科研，零基础如何入门

对于想转行的同学，或者是单纯的喜欢激光SLAM同学，不需要写论文，该怎么入门呢？

基本上是一样的，只是第六步可以忽略，可以不用读论文．如果不需要改代码的话第十步也是可以忽略的．而第九步，阅读源码，这一部分可以进行扩展，多阅读几个SLAM算法的源码与实现，将有助于帮你更好的理解SLAM的实现．

总结

这一路走下来真是不容易啊，涉及到了很多元的知识，也需要花挺长的时间来探索，来入门．如果没人带路的话可能要花更久的时间来找到正确的路，希望上边的步骤可以帮到你们．

这只是我的一点建议，不一定正确，只是给各位小白们提供个参考．

上边的步骤有很多是可以同时进行的，您可以自己选择如何组合．

上边的路线还有很多知识点没有提到，如 ROS 中的TF，这是所有SLAM中都会用到的；如 SLAM的理论求解算法，像高斯牛顿，LM等等．这篇文章没有讲太多理论算法方面的东西，以对小白更加友好．

# data[].attributes.attachment_link.keywords[]
同时定位和地图构建（slam）

# data[].attributes.attachment_link.keywords[]
机器人操作平台 (ros)

# data[].attributes.attachment_link.title
零基础如何入门激光SLAM

# data[].attributes.attachment_link.description
最近有几个人加我，都说是刚开始学激光slam，基本都是研一，也有一些大四的（大四的都开始学SLAM了！）。 情况也都差不多，有的是课题组里只有自己做slam，有的是完全没人带，有的是完全没基础，c++都没学过的那种。 这些小白都不知道怎么进行学习，问了我好多相同的问题，所以我就将我自己的思路整理了一下，也就是这篇文章的由来． ...

# data[].attributes.attachment_link.readable_text
C++基本語法

基本程式設計邏輯

解決大量資料問題

電腦擁有超快速且準確的運算能力，目前的電腦或智慧型手機可以在一秒內計算數十億次四則運算，協助了現代人類在日常生活、研究、商業...各個情境裡所遇到各式各樣的問題。學會命令電腦做事（即程式設計），就等於擁有一位忠心、絕對服從且威力強大的僕人。

但這位僕人相當的一板一眼，如果你對他下達了錯誤的指令，它也會毫不猶豫地迅速完成。若是主人下命令的邏輯不好，僕人做出來的事也會與主人的目標天差地遠。所以就算擁有電腦這位僕人，你還是必須掌握如何準確命令它達成你想要的目標。因此在這門課結束後，你的邏輯能力將會大幅上升、能更有系統性的思考問題的解決方法。

若是你的學習成果卓越，可以挑戰每年都有舉辦的International Olympiad in Informatics（IOI，國際高中資訊奧林匹亞），我國每年會派出四名國手參加此競賽。在邁向國手的路程中，若進入到台灣資訊奧林匹亞選手培訓營第二階段（每年約10名），就能透過推薦入學管道直接進入台、清、交等電機資訊領域之科系。

連結

IOI官方網站　 http://www.ioinformatics.org/index.shtml

# data[].attributes.attachment_link.description
C++基本語法 基本程式設計邏輯 解決大量資料問題 　　電腦擁有超快速且準確的運算能力，目前的電腦或智慧型手機可以在一秒內計算數十億次四則運算，協助了現代人類在日常生活、研究、商業...各個情境裡所遇到各式各樣的問題。學會命令電腦做事（即程式設計），就等於擁有一位忠心、絕對服從且威力強大的僕人。 ...

# data[].attributes.attachment_link.title
Mind Map Free - Free Mind Mapping Software | MindMeister

# data[].attributes.attachment_link.title
Mind Map Free - Free Mind Mapping Software | MindMeister

# data[].attributes.attachment_link.title
GitHub - WongKinYiu/yolov7: Implementation of paper - YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors

# data[].attributes.attachment_link.description
Implementation of paper - YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors MS COCO Docker environment (recommended) Expand # create the docker container, you can change the share memory size if you have more.

# data[].attributes.attachment_link.readable_text
Image by Author

Streamlit is a Python framework with which developers can quickly build web apps without frontend coding. On top of it, developers can make real-time video/audio processing apps that receive video/audio streams from users' media devices, only with ~10 lines of code in the case of the simplest example.

Since such apps are web-based, they can be deployed to the cloud, shared with users easily, and have modern and user-friendly UIs.

This tech stack is useful for creating demos and prototyping ideas of video/audio apps such as human or object detection, style transfer, image filters, speech recognition, video chat apps, and more.

A sample web-based object detection app. Users can change the threshold interactively during execution. Online demo🎈

A sample web-based style transfer app. Users can change model type and model parameters interactively during execution. Online demo🎈

You can see more examples at the examples section below.

NOTE: These sample apps are hosted on the public cloud ( Streamlit Cloud), and the video and audio streams are transmitted to and processed at the cloud server. While those data are only processed on memory and not saved to any storage, however, if you are concerned, please do not use them. As for the following contents in this article, we can execute all of them on our local. In addition, you can try the examples above on your local by following the instructions at the examples section below.

We have been typically using OpenCV to build real-time demo apps of image or video processing. Some of you (especially developers or researchers in such fields) may have seen the following code or similar many times.

import cv2

cap = cv2.VideoCapture(0)

while True:
 ret, frame = cap.read()

 img = cv2.Canny(frame, 100, 200) # Some image processing

 cv2.imshow('frame', img)
 if cv2.waitKey(1) & 0xFF == ord('q'):
 break

cap.release()
cv2.destroyAllWindows()

Compared to the GUI apps like above using cv2.VideoCapture and cv2.imshow that run on local environments, web-based apps have some advantages like below.

If we deploy the apps on the cloud, we can share the apps with our users simply by sending the URLs.

The users can use the apps only by accessing them through web browsers. It does not require any set-ups or external dependencies.

Easy to share and run:

Because all the users need is web browsers, the users can use the apps on their smartphones. It's convenient if we can show demos on such portable devices.

Usable on smartphones:

Developers can use text inputs, sliders, or other web-based components to accept user inputs or show data. Such web-based UIs are more friendly for users than desktop GUIs in recent days.

User-friendly UIs:

We will create a simple web-based real-time video processing app with ~10 or 20 LoC. Please try this tutorial in an environment where a webcam and a microphone are available.

You can check the final result of this tutorial in this repository. The deployed online demo is here🎈

In this tutorial, we will write code in app.py. Please create an empty app.py at first.

$ touch app.py

Install necessary packages

Next, we have to install the packages necessary for this tutorial.

streamlit: The Streamlit main package.

streamlit-webrtc: A custom component of Streamlit which deals with real-time video and audio streams.

opencv-python-headless: OpenCV. We choose the headless version here because we will construct the UI with Streamlit.

$ pip install -U streamlit streamlit-webrtc opencv-python-headless

First contact with Streamlit

NOTE: Please skip this section if you have experience in Streamlit.

First of all, launch the Streamlit with the command below. Please run the command in the same directory to the app.py.

$ streamlit run app.py

After a while, the Streamlit server process will boot up. Then access http://localhost:8501 to see the page like below (or it will automatically open in the browser by default). The screenshot here is in dark-mode, and if you are using light-mode, it looks different.

At this moment, there is no content on the web page because app.py is empty. We will add lines of code in the app.py for the Streamlit app.

Open the app.py with your editor and write the code below.

import streamlit as st

st.title("My first Streamlit app")
st.write("Hello, world")

When you save the file, Streamlit will detect the file change and shows the "Rerun" and "Always rerun" buttons on the top right of the screen.

Click the "Rerun" button. Then the web page is reloaded, and the page content would be like below. The web page content is generated based on the app.py code.

If you had clicked the "Always rerun" button, the page would automatically be reloaded every time the file changed.

Note that you have to reload the page like above in the following instructions where you update the app.py.

Now we have walked through the basic development flow of Streamlit apps. You write Python code with Streamlit components like st.title() and st.write() and pass it to the streamlit run command, then Streamlit generates the corresponding frontend contents on the web page.

In the next section, we will see how to develop a real-time video processing app on top of Streamlit. Apart from that, Streamlit itself covers more use cases such as machine learning, data science, or more general purposes. For such use cases, please see the official Streamlit tutorial for example.

Introduce the real-time video/audio streaming component

Update the app.py as below.

import streamlit as st
from streamlit_webrtc import webrtc_streamer

st.title("My first Streamlit app")
st.write("Hello, world")

webrtc_streamer(key="example")

We have added a single line with webrtc_streamer(). The web app would be like the screenshot below.

At the first trial, it may take some time to compile the package so that the page keeps showing the "running" message for a while after clicking the "Rerun" button. In such a case, wait for the process to finish.

Click the "START" button to start the video and audio streaming. You may be asked for permission to access the webcam and microphone at the first trial. Grant permission in that case.

The webrtc_streamer(key="example") above is a Streamlit component which deals with video and audio real-time I/O through web browsers. The key argument is a unique ID in the script to identify the component instance. We have set it as "example" here, but you can use any string for it. The component in this example only receives video and audio from the client-side webcam and microphone and outputs the raw streams. It's the most basic version of the component. We are going to enhance its functionality by adding other options in the following sections.

Development of a real-time video processing application

Update the app.py as follows.

import streamlit as st
from streamlit_webrtc import webrtc_streamer
import av
import cv2

st.title("My first Streamlit app")
st.write("Hello, world")


class VideoProcessor:
 def recv(self, frame):
 img = frame.to_ndarray(format="bgr24")

 img = cv2.cvtColor(cv2.Canny(img, 100, 200), cv2.COLOR_GRAY2BGR)

 return av.VideoFrame.from_ndarray(img, format="bgr24")


webrtc_streamer(key="example", video_processor_factory=VideoProcessor)

Try it out by clicking the "START" button like the previous section. With this new example, you can find that an image filter is applied to the video stream.

We have defined a class and its method VideoProcessor#recv(), a callback that receives an input frame and returns an output frame. We also put image processing (edge detection in this example) code inside the callback. As a result, we have injected the image processing code into the real-time video app through the callback.

webrtc_streamer() can take a class object with .recv() method as its video_processor_factory argument.

The .recv() receives and returns input and output image frames. These are instances of the VideoFrame class from . The PyAV library is a Python binding of ffmpeg, which provides video and audio capabilities. It is installed as a dependency of streamlit-webrtc.

The argument of .recv() is an image frame in the input video stream sourced from the webcam. It can be converted into a NumPy array with frame.to_ndarray().

The returned value from .recv() is displayed on the screen. In the sample above, a new VideoFrame object to be returned is generated from a NumPy array, with av.VideoFrame.from_ndarray(img, format="bgr24").

Any code can be put inside the callback. In the example above, we have used an edge detection filter cv2.Canny(img, 100, 200) (and a grayscale converter cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)) as an example.

Detailed explanations about the code follow.

Now, we have created a browser-ready real-time video processing app! We used a simple Canny edge detector in this example, and you can replace it with any image processing code in your original app.

If we use object detection or style transfer for that part, the app would be like the screenshots at the beginning of this article.

Receive user inputs

Update the app.py as below.

import streamlit as st
from streamlit_webrtc import webrtc_streamer
import av
import cv2

st.title("My first Streamlit app")
st.write("Hello, world")


class VideoProcessor:
 def __init__(self) -> None:
 self.threshold1 = 100
 self.threshold2 = 200

 def recv(self, frame):
 img = frame.to_ndarray(format="bgr24")

 img = cv2.cvtColor(cv2.Canny(img, self.threshold1, self.threshold2), cv2.COLOR_GRAY2BGR)

 return av.VideoFrame.from_ndarray(img, format="bgr24")


ctx = webrtc_streamer(key="example", video_processor_factory=VideoProcessor)
if ctx.video_processor:
 ctx.video_processor.threshold1 = st.slider("Threshold1", min_value=0, max_value=1000, step=1, value=100)
 ctx.video_processor.threshold2 = st.slider("Threshold2", min_value=0, max_value=1000, step=1, value=200)

Then click the "START" button. You will find that there are 2 sliders in this example. You can modify the parameters of cv2.Canny() through the sliders, even during execution in real time.

We added .threshold1 and .threshold2 attributes to the VideoProcessor class. They are used as parameters of cv2.Canny() inside the .recv().

webrtc_streamer returns a context object, so we assign it to the ctx variable.

A class object passed to webrtc_streamer() via the video_processor_factory argument will be instantiated and set to ctx.video_processor. It is not set when the video stream is not active, so we check its existence with the if clause.

So, in this example, the instance of VideoProcessor class will be set to ctx.video_processor, and we can access its .threshold1 and .threshold2 attributes. Then we assign the slider values to these attributes so that the parameters of cv2.Canny() can be changed via the sliders. I will explain later why we don't (can't) pass the slider values to cv2.Canny() directly.

The slider UI is created with st.slider() in Streamlit apps. st.slider() is a built-in Streamlit component. Its official API reference is https://docs.streamlit.io/library/api-reference/widgets/st.slider.

The explanations of the code follow:

Execution model of recv() and an important notice about it

Unlike OpenCV, streamlit-webrtc requires callbacks to process image and audio frames. The callback.recv() is one major difference between OpenCV GUI and streamlit-webrtc, and there are a few things you have to be aware of about it.

The global keyword does not work as expected inside .recv().

Streamlit methods such as st.write() cannot be used inside .recv().

Communications between inside and outside .recv() must be thread-safe.

Please note that .recv() is executed in a forked thread different from the main thread where the Streamlit app code runs. It makes some restrictions as below.

In the previous example, we passed values from the global scope to .recv() through the instance attributes since we cannot use global and refer to these values from inside .recv().

We are going to make the web app available to everyone by deploying it to the cloud.

Configure WebRTC

To deploy the app to the cloud, we have to add rtc_configuration parameter to the webrtc_streamer().

ctx = webrtc_streamer(
 key="example",
 video_processor_factory=VideoProcessor,
 rtc_configuration={ # Add this line
 "iceServers": [{"urls": ["stun:stun.l.google.com:19302"]}]
 }
)

This configuration is necessary to establish the media streaming connection when the server is on a remote host.

streamlit_webrtc uses WebRTC for its video and audio streaming. It has to access a "STUN server" in the global network for the remote peers (precisely, peers over the NATs) to establish WebRTC connections. As we don't see the details about STUN servers in this article, please google it if interested with keywords such as STUN, TURN, or NAT traversal.

We configured the code to use a free STUN server provided by Google in the example above. You can also use any other available STUN servers.

The value of the rtc_configuration argument will be passed to the RTCPeerConnection constructor on the frontend.

HTTPS

We have to serve web apps on remote hosts via HTTPS to use webcams or microphones.

Not only the webrtc_streamer() component we used here but also any frontend apps that access the client-side webcams or microphones use MediaDevices.getUserMedia() API. This API does not work in an "insecure context."

The document says

A secure context is, in short, a page loaded using HTTPS or the MediaDevices.getUserMedia() - Privacy and security URL scheme, or a page loaded from .

As a result, we need HTTPS to serve web apps on remote hosts which access the client-side webcams or microphones.

Streamlit Cloud

I recommend Streamlit Cloud for Streamlit app hosting. You can deploy the apps from GitHub repositories with a few clicks, and it automatically serves the apps via HTTPS. And Streamlit Cloud seems to provide better runtime than Heroku free-tier, while Streamlit Cloud provides a large deployment capacity for free.

Please refer to the official document for its usage.

I actually deployed the app we have seen in this article on Streamlit Cloud: https://share.streamlit.io/whitphx/streamlit-webrtc-article-tutorial-sample/main/app.py .

Its GitHub repository is https://github.com/whitphx/streamlit-webrtc-article-tutorial-sample .

Note that requirements.txt has been added to install the necessary dependencies ( streamlit-webrtc and opencv-python-headless) in the Streamlit Cloud environment: https://github.com/whitphx/streamlit-webrtc-article-tutorial-sample/blob/main/requirements.txt

As written above, the video and audio streams sourced from the client devices are transmitted to and processed at the server.

So, this library is not scalable and depends on network connectivity. You may think of it mainly for prototyping or demo purpose.

You also have to consider hosting the apps in local networks if there are concerns about transmitting media to the remote cloud server.

This section is a copy of the sample list at https://github.com/whitphx/streamlit-webrtc.

Showcase including following examples and more

Object detection (This is the sample app a screenshot of which is at the beginning of this article)

OpenCV filter

Uni-directional video streaming

Audio processing

⚡️Repository, 🎈Online demo

You can try out this sample app using the following commands on your local env.

$ pip install streamlit-webrtc opencv-python-headless matplotlib pydub
$ streamlit run https://raw.githubusercontent.com/whitphx/streamlit-webrtc-example/main/app.py

Real-time Speech-to-Text

⚡️Repository, 🎈Online demo

It converts your voice into text in real time. This app is self-contained; it does not depend on any external API.

Real-time video style transfer

⚡️Repository, 🎈Online demo

It applies a wide variety of style transfer filters to real-time video streams.

Video chat

⚡️Repository (Online demo not available)

You can create video chat apps with ~100 lines of Python code.

Tokyo 2020 Pictogram

⚡️Repository: 🎈Online demo

MediaPipe is used for pose estimation.

You can deal with audio streams in a similar way as video. If you define a class with .recv() method and pass it to the audio_processor_factory argument, the callback will be executed with audio frames. In the case of audio, the input argument and the returned value of the callback are instances of the AudioFrame class.

Please see the source code of a sample app changing the audio gain or the Speech-to-Text app in the examples above.

# data[].attributes.attachment_link.title
Developing Web-Based Real-Time Video/Audio Processing Apps Quickly with Streamlit

# data[].attributes.attachment_link.description
Streamlit is a Python framework with which developers can quickly build web apps without frontend coding. On top of it, developers can make real-time video/audio processing apps that receive video/audio streams from users' media devices, only with ~10 lines of code in the case of the simplest example.

# data[].attributes.attachment_link.title
GitHub - whitphx/streamlit-webrtc: Real-time video and audio streams over the network, with Streamlit.

# data[].attributes.attachment_link.description
Handling and transmitting real-time video/audio streams over the network with Streamlit Object detection OpenCV filter Uni-directional video streaming Audio processing It converts your voice into text in real time. This app is self-contained; it does not depend on any external API.

# data[].attributes.attachment_link.description
Python implementation to stream camera feed from OpenCV videoCapture via RTSP server using GStreamer 1.0. This implementation has been developed and tested on Ubuntu 16.04 and 18.04. So the installation steps are specific to debian based linux distros.

# data[].attributes.attachment_link.description
Instantly share code, notes, and snippets. Python RTSP stream using OpenCV You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or window. Reload to refresh your session. Reload to refresh your session.

# data[].attributes.attachment_link.readable_text
In this article, we will see how we can create browser-ready real-time video/audio processing apps with Streamlit.

Streamlit is a Python framework with which developers can quickly build web apps without frontend coding. On top of it, developers can make real-time video/audio processing apps that receive video/audio streams from users' media devices, only with ~10 lines of code in the case of the simplest example.

Since such apps are web-based, they can be deployed to the cloud, shared with users easily, and have modern and user-friendly UIs.

This tech stack is useful for creating demos and prototyping ideas of video/audio apps such as human or object detection, style transfer, image filters, speech recognition, video chat apps, and more.

A sample web-based object detection app. Users can change the threshold interactively during execution. Online demo🎈

A sample web-based style transfer app. Users can change model type and model parameters interactively during execution. Online demo🎈

You can see more examples at the examples section.

NOTE: These sample apps are hosted on the public cloud ( Streamlit Cloud), and the video and audio streams are transmitted to and processed at the cloud server. While those data are only processed on memory and not saved to any storage, however, if you are concerned, please do not use them. As for the following contents in this article, we can execute all of them on our local. In addition, you can try the examples above on your local by following the instructions at the examples section.

We have been typically using OpenCV to build real-time demo apps of image or video processing. Some of you (especially developers or researchers in such fields) may have seen the following code or similar many times.

Compared to the GUI apps like above using cv2.VideoCapture and cv2.imshow that run on local environments, web-based apps have some advantages like below.

Easy to share and run:

If we deploy the apps on the cloud, we can share the apps with our users simply by sending the URLs.

The users can use the apps only by accessing them through web browsers. It does not require any set-ups or external dependencies.

Usable on smartphones:
Because all the users need is web browsers, the users can use the apps on their smartphones. It's convenient if we can show demos on such portable devices.

User-friendly UIs:
Developers can use text inputs, sliders, or other web-based components to accept user inputs or show data. Such web-based UIs are more friendly for users than desktop GUIs in recent days.

We will create a simple web-based real-time video processing app with ~10 or 20 LoC. Please try this tutorial in an environment where a webcam and a microphone are available.

You can check the final result of this tutorial in this repository. The deployed online demo is here🎈

In this tutorial, we will write code in app.py. Please create an empty app.py at first.

Next, we have to install the packages necessary for this tutorial.

$ pip install -U streamlit streamlit-webrtc opencv-python-headless

streamlit: The Streamlit main package.

streamlit-webrtc: A custom component of Streamlit which deals with real-time video and audio streams.

opencv-python-headless: OpenCV. We choose the headless version here because we will construct the UI with Streamlit.

NOTE: Please skip this section if you have experience in Streamlit.

First of all, launch the Streamlit with the command below. Please run the command in the same directory to the app.py.

After a while, the Streamlit server process will boot up. Then access http://localhost:8501 to see the page like below (or it will automatically open in the browser by default). The screenshot here is in dark-mode, and if you are using light-mode, it looks different.

At this moment, there is no content on the web page because app.py is empty. We will add lines of code in the app.py for the Streamlit app.

Open the app.py with your editor and write the code below.

import streamlit as st st.title( "My first Streamlit app" ) st.write( "Hello, world" )

When you save the file, Streamlit will detect the file change and shows the "Rerun" and "Always rerun" buttons on the top right of the screen.

Click the "Rerun" button. Then the web page is reloaded, and the page content would be like below. The web page content is generated based on the app.py code.

If you had clicked the "Always rerun" button, the page would automatically be reloaded every time the file changed.

Note that you have to reload the page like above in the following instructions where you update the app.py.

Now we have walked through the basic development flow of Streamlit apps. You write Python code with Streamlit components like st.title() and st.write() and pass it to the streamlit run command, then Streamlit generates the corresponding frontend contents on the web page.

In the next section, we will see how to develop a real-time video processing app on top of Streamlit. Apart from that, Streamlit itself covers more use cases such as machine learning, data science, or more general purposes. For such use cases, please see the official Streamlit tutorial for example.

Update the app.py as below.

We have added a single line with webrtc_streamer(). The web app would be like the screenshot below.

At the first trial, it may take some time to compile the package so that the page keeps showing the "running" message for a while after clicking the "Rerun" button. In such a case, wait for the process to finish.

Click the "START" button to start the video and audio streaming. You may be asked for permission to access the webcam and microphone at the first trial. Grant permission in that case.

The webrtc_streamer(key="example") above is a Streamlit component which deals with video and audio real-time I/O through web browsers. The key argument is a unique ID in the script to identify the component instance. We have set it as "example" here, but you can use any string for it. The component in this example only receives video and audio from the client-side webcam and microphone and outputs the raw streams. It's the most basic version of the component. We are going to enhance its functionality by adding other options in the following sections.

Update the app.py as follows.

Try it out by clicking the "START" button like the previous section. With this new example, you can find that an image filter is applied to the video stream.

We have defined a class and its method VideoProcessor#recv(), a callback that receives an input frame and returns an output frame. We also put image processing (edge detection in this example) code inside the callback. As a result, we have injected the image processing code into the real-time video app through the callback.

Detailed explanations about the code follow.

webrtc_streamer() can take a class object with .recv() method as its video_processor_factory argument.

The .recv() receives and returns input and output image frames. These are instances of the VideoFrame class from PyAV.
The PyAV library is a Python binding of ffmpeg, which provides video and audio capabilities. It is installed as a dependency of streamlit-webrtc.

The argument of .recv() is an image frame in the input video stream sourced from the webcam. It can be converted into a NumPy array with frame.to_ndarray().

The returned value from .recv() is displayed on the screen. In the sample above, a new VideoFrame object to be returned is generated from a NumPy array, with av.VideoFrame.from_ndarray(img, format="bgr24").

Any code can be put inside the callback. In the example above, we have used an edge detection filter cv2.Canny(img, 100, 200) (and a grayscale converter cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)) as an example.

Now, we have created a browser-ready real-time video processing app! We used a simple Canny edge detector in this example, and you can replace it with any image processing code in your original app.

If we use object detection or style transfer for that part, the app would be like the screenshots at the beginning of this article.

Update the app.py as below.

Then click the "START" button. You will find that there are 2 sliders in this example. You can modify the parameters of cv2.Canny() through the sliders, even during execution in real time.

The explanations of the code follow:

We added .threshold1 and .threshold2 attributes to the VideoProcessor class. They are used as parameters of cv2.Canny() inside the .recv().

webrtc_streamer returns a context object, so we assign it to the ctx variable.

A class object passed to webrtc_streamer() via the video_processor_factory argument will be instantiated and set to ctx.video_processor.
It is not set when the video stream is not active, so we check its existence with the if clause.

So, in this example, the instance of VideoProcessor class will be set to ctx.video_processor, and we can access its .threshold1 and .threshold2 attributes. Then we assign the slider values to these attributes so that the parameters of cv2.Canny() can be changed via the sliders.
I will explain later why we don't (can't) pass the slider values to cv2.Canny() directly.

The slider UI is created with st.slider() in Streamlit apps.

Execution model of recv() and an important notice about it

Unlike OpenCV, streamlit-webrtc requires callbacks to process image and audio frames. The callback.recv() is one major difference between OpenCV GUI and streamlit-webrtc, and there are a few things you have to be aware of about it.

Please note that .recv() is executed in a forked thread different from the main thread where the Streamlit app code runs. It makes some restrictions as below.

The global keyword does not work as expected inside .recv().

Streamlit methods such as st.write() cannot be used inside .recv().

Communications between inside and outside .recv() must be thread-safe.

In the previous example, we passed values from the global scope to .recv() through the instance attributes since we cannot use global and refer to these values from inside .recv().

We are going to make the web app available to everyone by deploying it to the cloud.

To deploy the app to the cloud, we have to add rtc_configuration parameter to the webrtc_streamer().

This configuration is necessary to establish the media streaming connection when the server is on a remote host.

streamlit_webrtc uses WebRTC for its video and audio streaming. It has to access a "STUN server" in the global network for the remote peers (precisely, peers over the NATs) to establish WebRTC connections. As we don't see the details about STUN servers in this article, please google it if interested with keywords such as STUN, TURN, or NAT traversal.

We configured the code to use a free STUN server provided by Google in the example above. You can also use any other available STUN servers.

The value of the rtc_configuration argument will be passed to the RTCPeerConnection constructor on the frontend.

We have to serve web apps on remote hosts via HTTPS to use webcams or microphones.

Not only the webrtc_streamer() component we used here but also any frontend apps that access the client-side webcams or microphones use MediaDevices.getUserMedia() API. This API does not work in an "insecure context."

The document says

A secure context is, in short, a page loaded using HTTPS or the file:/// URL scheme, or a page loaded from localhost. MediaDevices.getUserMedia() - Privacy and security

As a result, we need HTTPS to serve web apps on remote hosts which access the client-side webcams or microphones.

I recommend Streamlit Cloud for Streamlit app hosting. You can deploy the apps from GitHub repositories with a few clicks, and it automatically serves the apps via HTTPS. And Streamlit Cloud seems to provide better runtime than Heroku free-tier, while Streamlit Cloud provides a large deployment capacity for free.

Please refer to the official document for its usage.

I actually deployed the app we have seen in this article on Streamlit Cloud: https://share.streamlit.io/whitphx/streamlit-webrtc-article-tutorial-sample/main/app.py .

Its GitHub repository is https://github.com/whitphx/streamlit-webrtc-article-tutorial-sample .

Note that requirements.txt has been added to install the necessary dependencies ( streamlit-webrtc and opencv-python-headless) in the Streamlit Cloud environment: https://github.com/whitphx/streamlit-webrtc-article-tutorial-sample/blob/main/requirements.txt

As written above, the video and audio streams sourced from the client devices are transmitted to and processed at the server.

So, this library is not scalable and depends on network connectivity. You may think of it mainly for prototyping or demo purpose.

You also have to consider hosting the apps in local networks if there are concerns about transmitting media to the remote cloud server.

This section is a copy of the sample list at https://github.com/whitphx/streamlit-webrtc.

⚡️Repository, 🎈Online demo

Object detection (This is the sample app a screenshot of which is at the beginning of this article)

OpenCV filter

Uni-directional video streaming

Audio processing

You can try out this sample app using the following commands on your local env.

$ pip install streamlit-webrtc opencv-python-headless matplotlib pydub $ streamlit run https://raw.githubusercontent.com/whitphx/streamlit-webrtc-example/main/app.py

⚡️Repository, 🎈Online demo

It converts your voice into text in real time. This app is self-contained; it does not depend on any external API.

⚡️Repository, 🎈Online demo

It applies a wide variety of style transfer filters to real-time video streams.

⚡️Repository (Online demo not available)

You can create video chat apps with ~100 lines of Python code.

⚡️Repository: 🎈Online demo

MediaPipe is used for pose estimation.

Pictogram app on web browsers.
After forking the original code which uses OpenCV, it takes only 2 hours to add the web UI supporting real time video streams with Streamlit and streamlit-webrtc. https://t.co/VRpB27enlI https://t.co/MQAWJVqkiM
- Yuichiro (@whitphx) August 2, 2021

You can deal with audio streams in a similar way as video. If you define a class with .recv() method and pass it to the audio_processor_factory argument, the callback will be executed with audio frames. In the case of audio, the input argument and the returned value of the callback are instances of the AudioFrame class.

Please see the source code of a sample app changing the audio gain or the Speech-to-Text app in the examples above.

# data[].attributes.attachment_link.title
Developing web-based real-time video/audio processing apps quickly with Streamlit

# data[].attributes.attachment_link.description
In this article, we will see how we can create browser-ready real-time video/audio processing apps with Streamlit. Streamlit is a Python framework with which developers can quickly build web apps without frontend coding. On top of it, developers can make real-time video/audio processing apps that receive video/audio streams from users' media devices, only with ~10 lines of code in the case of the simplest example.

# data[].attributes.attachment_link.title
GitHub - jeffbass/imagezmq: A set of Python classes that transport OpenCV images from one computer to another using PyZMQ messaging.

# data[].attributes.attachment_link.description
imageZMQ is a set of Python classes that transport OpenCV images from one computer to another using PyZMQ messaging. For example, here is a screen on a Mac computer showing simultaneous video streams from 8 Raspberry Pi cameras: Using imageZMQ, this is possible with 11 lines of Python on each Raspberry Pi and with 8 lines of Python on the Mac.

# data[].attributes.attachment_link.title
2022 Schedule of Weekly Paper Review Presentations