Publish Frames to S3 Storage in Vision AI Detection Apps#

Applications can take advantage of the S3 publish feature from DL Streamer Pipeline Server and use it to save frames to an S3 compatible storage.

Steps#

Note

For the purpose of this demonstration, we will be using SeaweedFS as the S3 storage. The necessary compose configuration for the SeaweedFS microservices is already part of the Docker Compose file.

Setup the application to use the docker based deployment following this document.
Bring up the containers.

Note

If you are running multiple instances of the application, start the services using ./run.sh up instead.
```
docker compose up -d
```
Install the package boto3 in your Python environment if not installed.

It is recommended to create a virtual environment and install it there. You can run the following commands to add the necessary dependencies as well as create and activate the environment.
```
sudo apt update && \
sudo apt install -y python3 python3-pip python3-venv
```
```
python3 -m venv venv && \
source venv/bin/activate
```
Once the environment is ready, install boto3 with the following command.
```
pip3 install --upgrade pip && \
pip3 install boto3==1.36.17
```
Note

DL Streamer Pipeline Server expects the bucket to be already present in the database. The next step will help you create one.

Create an S3 bucket using the following script.

Update the HOST_IP and credentials with that of the running SeaweedFS S3 server. Use create_bucket.py as the file name.

import boto3
url = "http://<HOST_IP>:<S3_STORAGE_HOST_PORT>"
user = "<value of S3_STORAGE_USERNAME used in .env>"
password = "<value of S3_STORAGE_PASSWORD used in .env>"
bucket_name = "ecgdemo"

client= boto3.client(
            "s3",
            endpoint_url=url,
            aws_access_key_id=user,
            aws_secret_access_key=password
)
client.create_bucket(Bucket=bucket_name)
buckets = client.list_buckets()
print("Buckets:", [b["Name"] for b in buckets.get("Buckets", [])])

Run the above script to create the bucket.

python3 create_bucket.py

Start the pipeline with the following cURL command with <HOST_IP> set to system IP. Ensure to give the correct path to the model as seen below. This example starts an AI pipeline.

Note

If you are running multiple instances of the application, ensure to provide NGINX_HTTPS_PORT number in the URL for the app instance, i.e., replace <HOST_IP> with <HOST_IP>:<NGINX_HTTPS_PORT> If you are running a single instance and using an NGINX_HTTPS_PORT other than the default 443, replace <HOST_IP> with <HOST_IP>:<NGINX_HTTPS_PORT>.

Pallet Defect Detection

curl -k https://<HOST_IP>/api/pipelines/user_defined_pipelines/pallet_defect_detection_s3write -X POST -H 'Content-Type: application/json' -d '{
   "source": {
       "uri": "file:///home/pipeline-server/resources/videos/warehouse.avi",
       "type": "uri"
   },
   "destination": {
       "frame": {
           "type": "webrtc",
           "peer-id": "pdds3",
           "overlay-properties": {
               "font-scale": 1.0,
               "draw-txt-bg": false
           }
       }
   },
   "parameters": {
       "detection-properties": {
           "model": "/home/pipeline-server/resources/models/pallet-defect-detection/deployment/Detection/model/model.xml",
           "device": "CPU"
       }
   }
}'

PCB Anomaly Detection

curl -k https://<HOST_IP>/api/pipelines/user_defined_pipelines/pcb_anomaly_detection_s3write -X POST -H 'Content-Type: application/json' -d '{
   "source": {
       "uri": "file:///home/pipeline-server/resources/videos/anomalib_pcb_test.avi",
       "type": "uri"
   },
   "destination": {
       "frame": {
           "type": "webrtc",
           "peer-id": "anomaly_s3",
           "overlay-properties": {
               "font-scale": 1.0,
               "draw-txt-bg": false
           }
       }
   },
   "parameters": {
       "classification-properties": {
           "model": "/home/pipeline-server/resources/models/pcb-anomaly-detection/deployment/Anomaly classification/model/model.xml",
           "device": "CPU"
       }
   }
}'

Go to https://<HOST_IP>/storage/buckets/ecgdemo/camera1/ to browse the frames stored in the ecgdemo bucket under the camera1 folder prefix (replace ecgdemo/camera1 with the bucket/folder prefix you used, if different). You will be prompted to log in with the S3_STORAGE_USERNAME/S3_STORAGE_PASSWORD credentials provided in .env file (HTTP Basic Auth).

Note

If you are running multiple instances of the application, ensure to provide NGINX_HTTPS_PORT number in the URL for the app instance, i.e., replace <HOST_IP> with <HOST_IP>:<NGINX_HTTPS_PORT> If you are running a single instance and using an NGINX_HTTPS_PORT other than the default 443, replace <HOST_IP> with <HOST_IP>:<NGINX_HTTPS_PORT>.