How to write a video player with iced?

Learn
1

I want to rewrite my video player from qml to iced. But I notice the CPU usage is very high. I don’t know how to handle it.

I implement it by gstreamer-rs => RGBA => iced Image. Is it the right way to use iced to play a video?

image
image2344×1062 291 KB

image
image2273×1070 268 KB

2

It’s my video implement.

use std::sync::{
    atomic::{AtomicBool, Ordering::Relaxed},
    Arc, RwLock,
};

use gstreamer::{
    prelude::{Cast, GstBinExtManual, ObjectExt},
    Caps, ClockTime, ElementFactory, FlowError, FlowSuccess, Pipeline,
    State::Playing,
};
use gstreamer_app::{prelude::ElementExt, AppSink, AppSinkCallbacks};
use iced::{
    subscription,
    widget::{image, Image},
    Length, Subscription,
};
use slog::Logger;
use tokio::sync::Semaphore;

use crate::{app::Message, return_error};

pub struct Video {
    log: Logger,
    id: usize,
    url: String,
    img: RwLock<Option<image::Handle>>,
    semaphore: Semaphore,
    closed: Arc<AtomicBool>, // TODO
}

impl Video {
    pub fn new(log: Logger, id: usize, url: String) -> Arc<Self> {
        let me = Arc::new(Video {
            log,
            id,
            url,
            img: RwLock::new(None),
            semaphore: Semaphore::new(0),
            closed: Arc::new(Default::default()),
        });

        let me_cloned = me.clone();
        tokio::spawn(async move {
            let res = me_cloned.clone().start_pipeline().await;
            return_error!(res, me_cloned.log, "pipeline error");
        });

        me
    }

    pub fn subscription(self: &Arc<Self>) -> Subscription<Message> {
        let self_cloned = self.clone();
        subscription::unfold(self.id, (), move |_| self_cloned.clone().read_video())
            .map(Message::VideoNewFrame) // FIXME
    }

    pub fn view(&self) -> Image<image::Handle> {
        let handle =
            self.img.read().unwrap().clone().unwrap_or_else(|| {
                image::Handle::from_memory(include_bytes!("../target/25231.png"))
            });
        Image::new(handle).width(Length::Fill).height(Length::Fill)
    }

    async fn read_video(self: Arc<Self>) -> ((), ()) {
        let _ = self.semaphore.acquire().await;
        ((), ())
    }

    async fn start_pipeline(self: Arc<Self>) -> anyhow::Result<()> {
        let src = ElementFactory::make("souphttpsrc").build()?;
        src.set_property("location", self.url.clone());
        src.set_property("retries", 0);

        let decoder = ElementFactory::make("jpegdec").build()?;

        let convert = ElementFactory::make("videoconvert").build()?;

        let caps = Caps::builder("video/x-raw").field("format", "RGBA").build();
        let capsfilter = ElementFactory::make("capsfilter")
            .property("caps", &caps)
            .build()?;

        let sink = AppSink::builder().build();

        let self_cloned = self.clone();
        let callbacks = AppSinkCallbacks::builder()
            .new_sample(move |sink| self_cloned.clone().new_sample(sink))
            .build();
        sink.set_callbacks(callbacks);

        let pipeline = Pipeline::default();
        pipeline.add_many([&src, &decoder, &convert, &capsfilter, sink.upcast_ref()])?;
        gstreamer::Element::link_many(&[&src, &decoder, &convert, &capsfilter, sink.upcast_ref()])?;
        pipeline.set_state(Playing)?;

        let bus = pipeline
            .bus()
            .ok_or(anyhow::format_err!("failed to get bus from pipeline"))?;
        for msg in bus.iter_timed(ClockTime::NONE) {
            match msg.view() {
                gstreamer::MessageView::Eos(_) => break,
                gstreamer::MessageView::Error(e) => {
                    return Err(anyhow::format_err!("{}", e.to_string()));
                }
                _ => {}
            }
        }

        Ok(())
    }

    fn new_sample(self: Arc<Self>, sink: &AppSink) -> Result<FlowSuccess, FlowError> {
        if self.closed.load(Relaxed) {
            return Err(FlowError::Eos);
        }

        let sample = sink.pull_sample().map_err(|_| FlowError::Eos)?;

        let buffer = sample.buffer().ok_or(FlowError::Error)?;

        let map = buffer.map_readable().map_err(|_| FlowError::Error)?;

        let caps = sample.caps().ok_or(FlowError::Error)?;
        let s = caps.structure(0).ok_or(FlowError::Error)?;
        let width = s.get::<i32>("width").map_err(|_| FlowError::Error)?;
        let height = s.get::<i32>("height").map_err(|_| FlowError::Error)?;
        // let format = s.get::<String>("format").map_err(|_| FlowError::Error)?;

        let handle = image::Handle::from_pixels(width as _, height as _, map.as_slice().to_vec());
        *self.img.write().unwrap() = Some(handle);

        self.semaphore.add_permits(1);

        Ok(FlowSuccess::Ok)
    }
}
3

Have youve seen this widget?

its pretty dead, but hopefully you can fork and fix it in 1/4 of the time.

4

Yes. I rewrite its minimal example to play 16 videos. But things are not getting better.

image
image1920×937 223 KB

5

I am unable to confirm that the G Streamer part in accurate, but I think the reason could be so CPU intensive is the amount of copies happening, but also the fact that image::Handle are not meant to be used this way.

Here not only is the data copied in iced, but then it also has to get copied to VRAM.
Another potential bottleneck is the that Handle::from_pixels hashes your images to identify it uniquely.

In short, I think iced_graphics does not provide an efficient way to render videos

I could be wrong I am no GPU programmer

6

OK. Thanks for your help. I plan to use github.com/KDAB/cxx-qt.

1 Like
7

post a crate if it’s working

8

It should be worked. I wrote it in qml before I used iced refactoring, I just need to complete the migration of C++ to Rust.

1 Like
9

@FH0 sorry only just saw this so hope it didnt waste your time but i have a working player.

feel free to do with it as you will. im currently in the process of implementing a ffmpeg backend on top of the gstreamer one which is proving to be quite annoying to do